protocol buffers - Upgrading protobuf from version 2 to 3 - incompatible with protobuf default values

Question

Im trying to upgrade to using protobuf version 3, and stay backwards compatible with version 2. Seems to work except for one thing - in proto-2 you could set your own default values, but in proto 3, you cant. If you chose a default value in proto-2 that is not the standard default value in proto-3, then you have a problem. For example, in proto-2:

message Record {
  required uint32 fileno = 1;               
  required uint64 pos = 2;                  
  optional uint64 bmsPos = 3 [default = 0]; 
  optional uint32 scanMode = 4 [default = 9999];  
}

now in proto-3 must be:

message Record {
  uint32 fileno = 1;               
  uint64 pos = 2;                  
  uint64 bmsPos = 3; 
  uint32 scanMode = 4;  
}

In both proto-2 and proto-3, missing values arent sent in the message. But the proto-3 API doesnt tell you if the default value is in the message or not, it just tells you the value.

So the proto-3 receiver gets a message and tells me that scanMode = 0. If that message came from a proto-2 sender, then either 1) the proto-2 sender placed a 0 in the message, or 2) the proto-2 sender set the value to 9999 (the default value), and so the value is not sent, and the proto-3 receiver interprets it as a 0. Without knowing if the value is present in the message or not, my code cant disambiguate, even if it knows whether the message came from a proto-2 or proto-3 sender.

Note that there's no problem with the bmsPos field in the example, since the proto-2 message uses the same default value as proto-3 (0). But if you happened to have chosen a default value not the same as proto-3, then i dont see how to upgrade to proto-3 and be backwards compatible.

question from:https://stackoverflow.com/questions/33204321/upgrading-protobuf-from-version-2-to-3-incompatible-with-protobuf-default-valu

Answer 1

Turns out there is a way to find out if a default value is actually missing or not (thanks to some friends at google for this answer):

message Record {
  uint32 fileno = 1;               
  uint64 pos = 2;                  
  uint64 bmsPos = 3; 
  oneof scanMode_present {
    uint32 scanMode = 4;
  }
  uint32 version = 5; // set to >= 3 for protobuf 3 
}

The generate code has additional methods to detect if oneof fields are set, using the getXXXcase() method:

int scanMode = proto.getScanMode();
boolean isMissing = proto.getScanModePresentCase() == Record.ScanModePresentCase.SCANMODEPRESENT_NOT_SET;
if (isMissing) {
  boolean isProto3 = proto.getVersion() >= 3;
  scanMode = (isProto3) ? 0 : 9999;
}

• Note that the name of the oneof is arbitrary, i have adopted the convention fieldname_present.
• The oneof doesnt add anything to the wire format, so it stays compatible with the proto-2 messages.
• You can add the version info anywhere that makes sense, i put it in the Record message for this example.

With this 'trick', i have upgraded to proto-3 with backwards compatibility with non-standard proto-2 default values.