Technical def for scx rpk (class) files

Technical part:  

I worked a lot to decode the data files (scx,rpk..) and I developed some proof-of-concept
(they do not have, how should I put it, proper production code quality
(if I would have done this at work, well let's just say code review would have been quite a shaming))
little programs using the knowledge, this includes: (available from the main SlrrExhaustive download)
  -Spl editor and map viewer (testing the concepts of RPK loading, and the spline .spl files define)
  -Extra Part aligner (this includes putting together the car as Slrr would through slot attachments)
  -UV editor (also, putting the car together properly so that the projected textures would line up)
  -The map editor can load maps in POLY or PHYS editor mode which can be used to move
    around vertices in a map or its collision
These programs are not made for general public use but I did not want to withhold
anything they may come in handy for someone someday.
The general model for the datafiles is usable the sourcecode gets problematic in the tools.
This was done in (old)c# because well it was easier, also all the sources are provided
(under the "Dont Be a Dick" Public License (https://www.dbad-license.org/)).




On this page I will provide the data file structures (as much as I found out)
which was used to implement the tools
(the c# source for datafile manipulation is understandable and serves as an example
implementation (not all exceptions and common errors are listed here but the c# source is tested)).
c# source for file manipulation

Scx files:
  There are two versions that Slrr uses for storing 3D model data:
  Scx v3 file structure (fields in order of appearance (in a valid file)):
    ScxV3
    {
      char[4] Signature = "INVO"
      int Version = 3
      ModelDefV3[] models //as many as can be also there may be an int
        at the end of the file that is 0
    }
    ModelDefV3
    {
      MaterialV3
      int vertexCount
      VertexDataV3 vertexData[vertexCount]
      int faceCount //triangle count 
      int IndicesV3[faceCount*3]
    }
    MaterialV3
    {
      int size
      
      float DiffuseColorR
      float DiffuseColorG
      float DiffuseColorB
      float Opacity
      
      float SpecularColorR
      float SpecularColorG
      float SpecularColorB
      float SpecularColorWeight
      float GlossinessWeight
      
      int Flags
      {
        int flagAlphaOpacity = 0x00000001;
        int flagDiffuseBlend = 0x00000100;
        int flagVertexColorBlend = 0x00000002;
        int flagLayer2VertexColorBlend = 0x00001000;
        int flagLayer2BlendByAlpha = 0x00000040;
      }
      
      short diffuseMapIndex //Which texture corresponds to which mapindex is set in the
        RenderRef definition part as a list of texture typeIDs in a RPK file referencing the given scx file
      short bumpMapIndex
      short specularMapIndex
      short reflectionMapIndex
      short diffuseLayer2MapIndex
      short unkownMapIndex2
      short illuminationIndex
      short unkownMapIndex3
      
      int oneVertexSize //in the next VertexDataV3
      
      int unkownInt1
      short unkownShort1
      
      short diffuseMixFirstMapChannel  //Channel is the index of UV that should be used 
      short diffuseMixSecondMapChannel
      short bumpMapChannel
      short specularMapChanel
      
      short unkownShort2
      int unkownInt2
      int unkownInt3
      int unkownInt4
      
      float illuminationColorR
      float illuminationColorG
      float illuminationColorB
      
      char materialName[32]
    }
    VertexDataV3// the size of this part is set in the MaterialV3 and can differ
      depending on the fields defined but the order of them is fixed so if an
      UV channel 3 should be defined all the field above must be as well
    {
      float vertexCoordX
      float vertexCoordY
      float vertexCoordZ
      float vertexNormalX
      float vertexNormalY
      float vertexNormalZ
      
      float uVChannel1X
      float uVChannel1Y
      float uVChannel2X
      float uVChannel2Y
      
      byte vertexColorB
      byte vertexColorG
      byte vertexColorR
      byte vertexColorA
      
      int unkown1
      int unkown2
      
      float uVChannel3X
      float uVChannel3Y
      
      int unkown3
    }
  Scx v4 file structure (fields in order of appearance (in a valid file)):
    ScxV4
    {
      char[4] Signature = "INVO"
      int Version = 4
      int headerEntriesCount
      HeaderEntry headerEntries[headerEntriesCount] // the order of these definitions
        implicitly declares the logical connection between entries
        (a mesh can only have one material definition...)
      byte data[] //header entries point into this
    }
    HeaderEntry
    {
      int entryType
      {
        FaceIndices = 5,  //FaceDef
        MaterialDef = 0,  //MaterialV4
        HardSurfaceVertexDefinition = 1,  //HardSurfaceDef
        VertexData = 4,  //VertexDataV4
        SparseBoneIndexList = 2,  //BoneList
        BoneIndexList = 3  //BoneList
      }
      int entryOffsetInFile
    }
    MaterialV4
    {
      int unkownZero = 0
      int size //of block
      int flags
      int unkownZero2 = 0
      int EntriesCount
      MaterialEntry entries[EntriesCount]
    }
    MaterialEntry
    {
      ushort unknownFlag
      ushort Type
      byte data[DataSize]
      //DataSize is dependant on the Type:
        //Type == 2048 => DataSize = 32 (char[32])
        //Type == 0 => DataSize = 4 (byte RGBA)
        //Type == 256 => DataSize = 4 (float intensity)
        //Type == 1536 => DataSize = 28 (MapDefinitionEntry)
        //Type == 1280 => DataSize = 4 (int)
      //You can also deduce semantic meaning
      (for this I used the TypeSwaped = (ushort)((Type >> 8) | (Type << 8)))
        //if (unknownFlag == 0 && TypeSwapped == 0)
        //  return "Diffuse Color";
        //if (unknownFlag == 2 && TypeSwapped == 0)
        //  return "Illumination Color";
        //if (unknownFlag == 1 && TypeSwapped == 0)
        //  return "Specular Color";
        //if (unknownFlag == 0 && TypeSwapped == 1)
        //  return "Glossiness";
        //if (unknownFlag == 1 && TypeSwapped == 1)
        //  return "Reflection Percent";
        //if (unknownFlag == 2 && TypeSwapped == 1)
        //  return "Bump Percent";
        //if (unknownFlag == 0 && TypeSwapped == 6)
        //  return "Diffuse Map";
        //if (unknownFlag == 1 && TypeSwapped == 6)
        //  return "Diffuse Mix Second";
        //if (unknownFlag == 2 && TypeSwapped == 6)
        //  return "Bump Map";
        //if (unknownFlag == 3 && TypeSwapped == 6)
        //  return "Reflection Map";
        //if (unknownFlag == 4 && TypeSwapped == 6)
        //  return "Illumination Map";
        //if (unknownFlag == 0 && TypeSwapped == 8)
        //  return "Material Name";
    }
    MapDefinitionEntry
    {
      int textureIndex  //this index is taken from the texture TypeID list defined
        in a RPK that references the scx file containing this map definition record
      int mapChannel  //UV used
      int TillingFlags//3=UTileVTile 0=UMirrorVMirror
      float TillingU
      float TillingV
      float OffsetU
      float OffsetV
      //if the next entry is a float intensity MaterailEntry it is the
        weight of this Map (like reflection strength)
    }
    FaceDef
    {
      int type
      int size //of block
      int numOfShorts
      ushort indices[numOfShorts]//I must point out that these are ushorts meaning if there are
        more than 0xffff (65535) vertices in a given vertexDataV4 indices can overflow which in
        turn can lead to crashes because of the extreme triangles
      //well actually slrr even used these as shorts when projecting decals onto parts
        (on collision or by painting) so there could only be 0xffff/2 vertices in a mesh
        (a model is made out of meshes)
      //technically:
        //004F2DA1 (offset in Slrr.exe): two loops looping through all indices and hence
          all vertices this can calculate bad pointers...
        //the loop checks a triangle (3 offsets are calculated but MOVSX is used to
          load EDI which is "singextend" meaning it is cast to a short from a ushort....
          this is very bad EDI can move backwards(even before the base pointer) in the buffer
          if the index is greater than MAX_USHORT/2 which can happen ... that's not that many vertices,
          all 6 (3 in each loop) MOVSX instructions should be MOVZX (zero extend))
    }
    VertexDataV4
    {
      int type
      int size //of block
      int numberOfVertices//will be indexed with ushorts in FaceDef so it should not be > 0xffff
      int vertexType//flags declaring which fields are defined in the VertexData
        //PossibleFeatures: (in order of possible appearance)
          //Position = 0x1, float[3] XYZ floats
          //BoneWeightNumIs0 = 0x2, 0 weight-float written (one is implicit 1-Sum(WrittenWeights) = 1)
          //BoneWeightNumIs1 = 0x4, 1 weight-float written (one is implicit 1-Sum(WrittenWeights))
          //BoneWeightNumIs2 = 0x8, 2 weight-floats written (one is implicit 1-Sum(WrittenWeights))
          //BoneWeightNumIs3 = 0x10, 3 weight-floats written (one is implicit 1-Sum(WrittenWeights))
          //BoneIndRef = 0x20, byte[4] (four indices to the BoneIndexList)
          //Normal = 0x40, float[3] XYZ floats
          //VertexIllumination = 0x80, byte[4] RGBA
          //VertexColor = 0x100, byte[4] RGBA
          //UV1 = 0x200, float[2] UV
          //UV2 = 0x400, float[2] UV
          //UV3 = 0x800, float[2] UV
          //BumpMapNormal = 0x40000, float[3] XYZ floats
      byte vertexData[(size - 16)] //vertex data defined in accordance with the vertexType
    }
    HardSurfaceDef //if this entry is present the current mesh is a hard surface mesh meaning
      there won"t be any bone references in the vertexDatas
    {
      int type
      int size
      int unkownInt = 0
    }
    BoneList
    {
      int type
      int size
      int unkownInt = 0
      int BoneIndexList[((size - (3 * 4)) / 4)]
    }
    
Rpk files:
  These files are used to define the relations between other data files
  (telling a Part script which cfg to use and which scx to use with which textures...)
  Rpk
  {
    Header header
    ExternalReference extRefs[header.externalReferencesCount]
    EntriesHeader entriesHeader[]
    ResEntry resEntries[entriesHeader.entriesCount]
    byte rsdData[]
  }
  Header
  {
    char signature[4] = "RPAK"
    int version512 = 512
    int externalReferencesCount
    int externalReferencesUnkownZero = 0
  }
  ExternalReference
  {
    short unkownIndexZero = 0
    short indexOfReference
    char referenceString[60] //padded with 0-s (at the end)
  }
  EntriesHeader
  {
    int entriesSize;//resEntries size in bytes
    int entriesCount
    int entriesType1Count // number of entries with typeOfEntry == 1
    int entriesNonType1Count
  }
  ResEntry
  {
    int superID
    int typeID// it is called Type ID because it only needs to be unique in
      combination with the value of typeOfEntry
    byte typeOfEntry
    byte additionalType
    float isParentCompatible
    int fileOffsetOfRSD
    int RSDLength
    byte aliasLength
    char alias[aliasLength] //despite the length defined this must be zeroterminated
      (size includes the zero)
    (//optional
     float floatBounds[12]// it is present if additionalType & 1 != 0
      this is used in the spatial data structure used to define maps
    )
  }
  RSDEntry
  {
    InnerRsdEntry innerEntries[]//as many as the corresponding RSDLength can hold
  }
  InnerRsdEntry can be one of; ICFGInnerEntry, XCFGInnerEntry, RSDInnerEntry,
  StringInnerEntry, EXTPInnerEntry, NamedDataArray, InnerPolyEntry, InnerPhysEntry
  ICFGInnerEntry //Internal cfg entry: a cfg file defined directly in the rpk
  {
    char signature[4] = "ICFG"
    int lengthOfData
    char cfgData[lengthOfData]// \0 delimited list of strings (2 strings observed)
      defining a common cfg with a key 
  }
  XCFGInnerEntry //reference cfg
  {
    char signature[4] = "XCFG"
    int lengthOfData
    char cfgData[lengthOfData]// \0 delimited list of strings the first one is usually
      empty the second one is a key, a whitespace and than a file path relative to the
      slrr root directory to a cfg file
  }
  RSDInnerEntry
  {
    char signature[4] = "RSD\0" 
    int lengthOfData
    char stringData[lengthOfData]// string data used mainly to define map "entry points"
      //expect NamedDatas and the spatial structure definition of a map (as in track not as in picture)
  }
  StringInnerEntry
  {
    char stringData[] //as much as the current RSD entry can hold
      (ResEntry.RSDLength (or as much as is left of it))
  }
  EXTPInnerEntry
  {
    char signature[4] = "EXTP" 
    int lengthOfData = 4
    int data//only 3 observed it is also the number of the root NamedDataArrays defined
      but that can be coincidence
  }
  NamedDataArray
  {
    char signature[4] = "\0\0\0\0" 
    int numOfNamedDatas
    NamedData namedDatas[numOfNamedDatas]//these represent the spatial tree data structure
      (not a BSP, some axis aligned thing not exactly an octree but definitely some bounding volume hierarchy)
  }
  NamedData
  {
    int TypeID
    int SuperID
    byte typeOfEntry
    byte additionalType
    float isParentCompatible
    int offset//in file, references NamedDataArray, InnerPolyEntry or InnerPhysEntry entries
      not always just one but uniquely; only one NamedData references a given array or poly or phys
    int sizeAtOffset
    byte nameLength
    char name[nameLength]//just as with the RES entries
    float boundingBoxX//in range -50000 .. 50000 positioning the centre of the box
    float boundingBoxY//in range -50000 .. 50000
    float boundingBoxZ//in range -50000 .. 50000
    float boundingBoxHalfWidthX//in range 0 .. 50000 generally small values
    float boundingBoxHalfWidthY//in range 0 .. 50000 generally small values
    float boundingBoxHalfWidthZ//in range 0 .. 50000 generally small values
    float unkownData7// only 0 seen
    float unkownData8// only 0 seen
    float unkownData9// only 0 seen
    float unkownData10// only 0 seen
    float unkownData11// only 0 seen
    float unkownData12// only 0 seen
  }
  InnerPolyEntry
  {
    char signature[4] = "POLY" 
    int size
    int unkownCount1
    int meshCount
    Mesh meshes[meshCount]
  }
  Mesh
  {
    int textureIndex
    int verticesCount
    int triangleCount
    VertexData vertexData[verticesCount]
    int Indices[triangleCount*3]
  }
  VertexData
  {
    float coordX
    float coordY
    float coordZ
    float normalX
    float normalY
    float normalZ
    byte colorR
    byte colorG
    byte colorB
    byte colorA
    byte illuminationR
    byte illuminationG
    byte illuminationB
    byte illuminationA
    float uVChannel1U
    float uVChannel1V
    float uVChannel2U
    float uVChannel2V
    float uVChannel3U
    float uVChannel3V
  }
  InnerPhysEntry
  {
    char signature[4] = "PHYS"
    int size
    int indexChunkCount
    IndexChunk indexChunks[indexChunkCount]
    int vertexChunkCount
    VertexChunk vertexChunks[vertexChunkCount]
  }
  IndexChunk
  {
    int unkownInt1//probably the material...
    int triIndices[3]
    float normalX
    float normalY
    float normalZ
  }
  VertexChunk
  {
    float vertexX
    float vertexY
    float vertexZ
  }
  
Class files:
  The compiled script files of slrr this uses a custom java implementation
  it was developed with Flex(,Bison) ... if someone wants a challenge (to investigate the saved AST)
  I only figured out the bare minimum that is needed to change the rpk references,
  class and package names in the .class files without .java sources
  Class
  {
    TUFA definedClasses[]//sequentially
  }
  TUFA
  {
    char signature[4] = "TUFA"
    int unkownData1
    int unkownData2
    SignaturedSizedScriptEntry entries[]//as many as can be till the next TUFA entry,
      can be SignaturedSizedScriptEntry or CONSEntry
  }
  SignaturedSizedScriptEntry
  {
    char signature[4]
    int size;
    byte data&#91;size]
  }
  CONSEntry
  {
    char signature[4] = "CONS"
    int size;
    int entriesCount
    ConstantEntry entries[entriesCount] //can be ConstantEntry, NameConstantEntry or RpkRefConstant,
      the full class name(including the package) is the first NameConstantEntry
      the full parent class name is the second
  }
  ConstantEntry
  {
    int ID 
    byte[] data// size is determined by the ID:
      //ID == 0 => size = (the first int in data)+4
      //ID == 4 => size = 4
      //ID == 7 or 3 or 5 => size = 8
      //data is generally an int[] referencing other ConstantEntries by their indices in the
        given constant table if flatted out it will give a function reference or other string
  }
  NameConstantEntry
  {
    int ID = 0
    int stringLength
    char name[stringLength]
  }
  RpkRefConstant
  {
    int ID = 3
    int RPKnameIndexInConstantTable//the index of a NameConstantEntry in the current
      CONS that will be the rpk referenced (the file path)
    int TypeIdInRPK
  }
  
  
  
  Other assorted info about Slrr:
  In the cfgs for vehicles and parts stockpart 0x00000001 line means that when creating
  a dummy model try and attach the part with typeID 0x00000001 (this is used when
  creating the cars roaming the city) creating a dummy car:
  dummycar = new GameRef();
  dummycar.create_native( map, new GameRef(dummyID), "0,-10000,0,0,0,0", "dummycar" );
  here stockpart references will be attached
    
  In the cfgs for vehicles and parts the texture 0x00000001 line means that the texture 0x00000001
  (which should be in the textureID list of the corresponding renderRef definition) will be replaced
  when painting is applied (or the setTexture method)
  this can be used to create parts with paintable reflection or illumination textures which is quite cool.
    
  Slrr manages RenderRefs competently but scripts are difficult for it this is the reason
  why in the demonstration "modded almost to hell" version of Slrr I created all the RenderRef lookup tables
  (to delay the loading of the corresponding classes);
  once a script is loaded (meaning an instance of one of the defined classes is created) it can not be unloaded
  and this is why slrr will always crash eventually; it is a 32bit application so it can only address 
  ~4Gb memory which will easily run out because every part has a unique class,
  of course as demonstrated in the videos this can be pushed out significantly
  (using the renderRef lookups), and this is why all the ExtraParts use the same script.
  
  The silent crashes (ones that do not produce a line in the error.log) are
  (if you can rule out the obvious missing resource thing and class redefinition
  (which can be easily checked using the file type model))
  almost always caused by
  the GC freeing up a script instance from under another script-thread,
  multi threading is very dangerous in Slrr
  (Miran's added little camera icon to the navigator
  (which is updated from a thread created by the Track class) was a frequent cause of crashes).
    
  The game uses standard fixed function pipeline directx9
  
  The .spl file spline row format (%LF is float %d is int) is:
  (%lf %lf %lf) (%lf %lf %lf) %lf %lf %d %lf %lf %d:
  (pos XYZ), (normal XYZ), splineWidth, targetVelocity, I do not know the rest
  the two ints are used as bools
  the list of first two vectors define a closed cubic Hermite spline.
    
  Slrr can not load more than 256 RPKs and there are at least 5 reserved slots that get filled by the exe
  itself and not by loading a library this modded version loads 248 (I think), and that is pretty much the maximum 
  possible.
    
 
1.   Posted by Marìo Alonzo Garcia Duque De Los Perros Lindos   2018-02-22 14:27    

Imma send you my research text file as anyway i'm not really going to do anything else with it anymore, and an old exe tool i got, i'm sure it will help you defining some of the unknown values.

For example, i think you didnt add(but i might've just not seen it) self-illumination percentage, self-illumination color and self-illumination texture, the backface culling option amd something else right now i just dont remember.

Anyway you'll see it by your own and eventually add stuff to your research


2.   Posted by HardFFlip   2018-02-22 16:18    

Any chance to make the scx v4 export plugin for newer versions of 3ds Max? 2015 or higher, for example. Because sometimes exporting from 3ds Max 2005 like pain in da butt.


3.   Posted by amilmand   2018-02-22 19:37    

Bigg Boss93:
illumination color
Illumination is there but if you can extend it surely more knowledge is always better.
HardFFlip:
v4 export plugin
Well the definition is there and can be used to make an exporter for someone familiar with the plugin architecture of 3dsmax (I would probably expect a blender plugin instead but who knows) I myself really doubt to be able to find the time to learn this (since not every aspect of the scx file format can be edited in 3dsmax, thinking about the flags of a material in particular, the plugin needs to define some custom modifier and/or material to properly set every property of the resulting scx)
Nevertheless I maintain that, although archaic, using 3dsmax5 and the original exporters are preventing a lot of issues exactly because of the limitations it holds the modder to, the avarege quality of mods is low enough with "only" needing to care to make the cfg, rpk, and java definitions without errors, if one would more easily be able to make bad meshes aswell whether because of a badly written exporter or simply because the limitations were stripped... I dont know, it seems to be a far too complicated system to integrate into already :/


Total : 3, on page: 3