Perform field initialization to defaults only when the field is skipped.

Avoids unnecessary initialization & unnecessary scan of
the pb_field_t array.

Runtime on Cortex-M3 -5%, code size +2%.

Could need some more testing with random field orders.
Have to write a tool to randomize fields in a message.

CHANGELOG

@@ -1,29 +1,3 @@
-nanopb-0.2.2
- Add support for extension fields (issue 17)
- Fix unknown fields in empty message (issue 78)
- Include the field tags in the generated .pb.h file.
- Add pb_decode_delimited and pb_encode_delimited wrapper functions (issue 74)
- Add a section in top of pb.h for changing compilation settings (issue 76)
- Documentation improvements (issues 12, 77 and others)
- Improved tests
-
-nanopb-0.2.1
- NOTE: The default callback function signature has changed.
-       If you don't want to update your code, define PB_OLD_CALLBACK_STYLE.
- 
- Change the callback function to use void** (issue 69)
- Add support for defining the nanopb options in a separate file (issue 12)
- Add support for packed structs in IAR and MSVC (in addition to GCC) (issue 66)
- Implement error message support for the encoder side (issue 7)
- Handle unterminated strings when encoding (issue 68)
- Fix bug with empty strings in repeated string callbacks (issue 73)
- Fix regression in 0.2.0 with optional callback fields (issue 70)
- Fix bugs with empty message types (issues 64, 65)
- Fix some compiler warnings on clang (issue 67)
- Some portability improvements (issues 60, 62)
- Various new generator options
- Improved tests
-
 nanopb-0.2.0
  NOTE: This release requires you to regenerate all .pb.c
        files. Files generated by older versions will not

docs/concepts.rst

@@ -10,40 +10,47 @@ The things outlined here are the underlying concepts of the nanopb design.
 
 Proto files
 ===========
-All Protocol Buffers implementations use .proto files to describe the message
-format. The point of these files is to be a portable interface description
-language.
+All Protocol Buffers implementations use .proto files to describe the message format.
+The point of these files is to be a portable interface description language.
 
 Compiling .proto files for nanopb
 ---------------------------------
-Nanopb uses the Google's protoc compiler to parse the .proto file, and then a
-python script to generate the C header and source code from it::
+Nanopb uses the Google's protoc compiler to parse the .proto file, and then a python script to generate the C header and source code from it::
 
     user@host:~$ protoc -omessage.pb message.proto
     user@host:~$ python ../generator/nanopb_generator.py message.pb
     Writing to message.h and message.c
     user@host:~$
 
-Modifying generator behaviour
------------------------------
-Using generator options, you can set maximum sizes for fields in order to
-allocate them statically. The preferred way to do this is to create an .options
-file with the same name as your .proto file::
+Compiling .proto files with nanopb options
+------------------------------------------
+Nanopb defines two extensions for message fields described in .proto files: *max_size* and *max_count*.
+These are the maximum size of a string and maximum count of items in an array::
 
-   # Foo.proto
-   message Foo {
-      required string name = 1;
-   }
+    required string name = 1 [(nanopb).max_size = 40];
+    repeated PhoneNumber phone = 4 [(nanopb).max_count = 5];
 
-::
+To use these extensions, you need to place an import statement in the beginning of the file::
 
-   # Foo.options
-   Foo.name max_size:16
+    import "nanopb.proto";
 
-For more information on this, see the `Proto file options`_ section in the
-reference manual.
+This file, in turn, requires the file *google/protobuf/descriptor.proto*. This is usually installed under */usr/include*. Therefore, to compile a .proto file which uses options, use a protoc command similar to::
+
+    protoc -I/usr/include -Inanopb/generator -I. -omessage.pb message.proto
+
+The options can be defined in file, message and field scopes::
+
+    option (nanopb_fileopt).max_size = 20; // File scope
+    message Message
+    {
+        option (nanopb_msgopt).max_size = 30; // Message scope
+        required string fieldsize = 1 [(nanopb).max_size = 40]; // Field scope
+    }
+
+It is also possible to give the options on command line, but then they will affect the whole file. For example::
+
+    user@host:~$ python ../generator/nanopb_generator.py -s 'max_size: 20' message.pb
 
-.. _`Proto file options`: reference.html#proto-file-options
 
 Streams
 =======
@@ -256,50 +263,6 @@ generates this field description array for the structure *Person_PhoneNumber*::
  };
 
 
-Extension fields
-================
-Protocol Buffers supports a concept of `extension fields`_, which are
-additional fields to a message, but defined outside the actual message.
-The definition can even be in a completely separate .proto file.
-
-The base message is declared as extensible by keyword *extensions* in
-the .proto file::
-
- message MyMessage {
-     .. fields ..
-     extensions 100 to 199;
- }
-
-For each extensible message, *nanopb_generator.py* declares an additional
-callback field called *extensions*. The field and associated datatype
-*pb_extension_t* forms a linked list of handlers. When an unknown field is
-encountered, the decoder calls each handler in turn until either one of them
-handles the field, or the list is exhausted.
-
-The actual extensions are declared using the *extend* keyword in the .proto,
-and are in the global namespace::
-
- extend MyMessage {
-     optional int32 myextension = 100;
- }
-
-For each extension, *nanopb_generator.py* creates a constant of type
-*pb_extension_type_t*. To link together the base message and the extension,
-you have to:
-
-1. Allocate storage for your field, matching the datatype in the .proto.
-   For example, for a *int32* field, you need a *int32_t* variable to store
-   the value.
-2. Create a *pb_extension_t* constant, with pointers to your variable and
-   to the generated *pb_extension_type_t*.
-3. Set the *message.extensions* pointer to point to the *pb_extension_t*.
-
-An example of this is available in *tests/test_encode_extensions.c* and
-*tests/test_decode_extensions.c*.
-
-.. _`extension fields`: https://developers.google.com/protocol-buffers/docs/proto#extensions
-
-
 Return values and error handling
 ================================
 

docs/index.rst

@@ -41,7 +41,7 @@ Features and limitations
 #) Allows specifying maximum size for strings and arrays, so that they can be allocated statically.
 #) No malloc needed: everything can be allocated statically or on the stack.
 #) You can use either encoder or decoder alone to cut the code size in half.
-#) Support for most protobuf features, including: all data types, nested submessages, default values, repeated and optional fields, packed arrays, extension fields.
+#) Support for most protobuf features, including: all data types, nested submessages, default values, repeated and optional fields, packed arrays.
 #) Callback mechanism for handling messages larger than can fit in available RAM.
 #) Extensive set of tests.
 
@@ -103,8 +103,6 @@ Nanopb should compile with most ansi-C compatible compilers. It however requires
 
 If these header files do not come with your compiler, you should be able to find suitable replacements online. Mostly the requirements are very simple, just a few basic functions and typedefs.
 
-Alternatively, you can define *PB_SYSTEM_HEADER*, which should be the name of a single header file including all the necessary definitions.
-
 Debugging and testing
 =====================
 Extensive unittests are included under the *tests* folder. Just type *make* there to run the tests.

docs/reference.rst

@@ -6,187 +6,31 @@ Nanopb: API reference
 
 .. contents ::
 
-
-
-
 Compilation options
 ===================
-The following options can be specified in one of two ways:
+The following options can be specified using -D switch given to the C compiler:
 
-1. Using the -D switch on the C compiler command line.
-2. By #defining them at the top of pb.h.
-
-You must have the same settings for the nanopb library and all code that
-includes pb.h.
-
-============================  ================================================
-__BIG_ENDIAN__                 Set this if your platform stores integers and
-                               floats in big-endian format. Mixed-endian
-                               systems (different layout for ints and floats)
-                               are currently not supported.
-NANOPB_INTERNALS               Set this to expose the field encoder functions
-                               that are hidden since nanopb-0.1.3.
-PB_MAX_REQUIRED_FIELDS         Maximum number of required fields to check for
-                               presence. Default value is 64. Increases stack
-                               usage 1 byte per every 8 fields. Compiler
-                               warning will tell if you need this.
-PB_FIELD_16BIT                 Add support for tag numbers > 255 and fields
-                               larger than 255 bytes or 255 array entries.
-                               Increases code size 3 bytes per each field.
-                               Compiler error will tell if you need this.
-PB_FIELD_32BIT                 Add support for tag numbers > 65535 and fields
-                               larger than 65535 bytes or 65535 array entries.
-                               Increases code size 9 bytes per each field.
-                               Compiler error will tell if you need this.
-PB_NO_ERRMSG                   Disables the support for error messages; only
-                               error information is the true/false return
-                               value. Decreases the code size by a few hundred
-                               bytes.
-PB_BUFFER_ONLY                 Disables the support for custom streams. Only
-                               supports encoding and decoding with memory
-                               buffers. Speeds up execution and decreases code
-                               size slightly.
-PB_OLD_CALLBACK_STYLE          Use the old function signature (void\* instead
-                               of void\*\*) for callback fields. This was the
+============================  ================================================================================================
+__BIG_ENDIAN__                 Set this if your platform stores integers and floats in big-endian format.
+                               Mixed-endian systems (different layout for ints and floats) are currently not supported.
+NANOPB_INTERNALS               Set this to expose the field encoder functions that are hidden since nanopb-0.1.3.
+PB_MAX_REQUIRED_FIELDS         Maximum number of required fields to check for presence. Default value is 64. Increases stack
+                               usage 1 byte per every 8 fields. Compiler warning will tell if you need this.
+PB_FIELD_16BIT                 Add support for tag numbers > 255 and fields larger than 255 bytes or 255 array entries.
+                               Increases code size 3 bytes per each field. Compiler error will tell if you need this.
+PB_FIELD_32BIT                 Add support for tag numbers > 65535 and fields larger than 65535 bytes or 65535 array entries.
+                               Increases code size 9 bytes per each field. Compiler error will tell if you need this.
+PB_NO_ERRMSG                   Disables the support for error messages; only error information is the true/false return value.
+                               Decreases the code size by a few hundred bytes.
+PB_BUFFER_ONLY                 Disables the support for custom streams. Only supports encoding to memory buffers.
+                               Speeds up execution and decreases code size slightly.
+PB_OLD_CALLBACK_STYLE          Use the old function signature (void\* instead of void\*\*) for callback fields. This was the
                                default until nanopb-0.2.1.
-PB_SYSTEM_HEADER               Replace the standard header files with a single
-                               header file. It should define all the required
-                               functions and typedefs listed on the
-                               `overview page`_. Value must include quotes,
-                               for example *#define PB_SYSTEM_HEADER "foo.h"*.
-============================  ================================================
-
-The PB_MAX_REQUIRED_FIELDS, PB_FIELD_16BIT and PB_FIELD_32BIT settings allow
-raising some datatype limits to suit larger messages. Their need is recognized
-automatically by C-preprocessor #if-directives in the generated .pb.h files.
-The default setting is to use the smallest datatypes (least resources used).
-
-.. _`overview page`: index.html#compiler-requirements
-
-
-Proto file options
-==================
-The generator behaviour can be adjusted using these options, defined in the
-'nanopb.proto' file in the generator folder:
-
-============================  ================================================
-max_size                       Allocated size for *bytes* and *string* fields.
-max_count                      Allocated number of entries in arrays
-                               (*repeated* fields).
-type                           Type of the generated field. Default value
-                               is *FT_DEFAULT*, which selects automatically.
-                               You can use *FT_CALLBACK*, *FT_STATIC* or
-                               *FT_IGNORE* to force a callback field, a static
-                               field or to completely ignore the field.
-long_names                     Prefix the enum name to the enum value in
-                               definitions, i.e. *EnumName_EnumValue*. Enabled
-                               by default.
-packed_struct                  Make the generated structures packed.
-                               NOTE: This cannot be used on CPUs that break
-                               on unaligned accesses to variables.
-============================  ================================================
-
-These options can be defined for the .proto files before they are converted
-using the nanopb-generatory.py. There are three ways to define the options:
-
-1. Using a separate .options file.
-   This is the preferred way as of nanopb-0.2.1, because it has the best
-   compatibility with other protobuf libraries.
-2. Defining the options on the command line of nanopb_generator.py.
-   This only makes sense for settings that apply to a whole file.
-3. Defining the options in the .proto file using the nanopb extensions.
-   This is the way used in nanopb-0.1, and will remain supported in the
-   future. It however sometimes causes trouble when using the .proto file
-   with other protobuf libraries.
-
-The effect of the options is the same no matter how they are given. The most
-common purpose is to define maximum size for string fields in order to
-statically allocate them.
-
-Defining the options in a .options file
----------------------------------------
-The preferred way to define options is to have a separate file
-'myproto.options' in the same directory as the 'myproto.proto'. ::
-
-    # myproto.proto
-    message MyMessage {
-        required string name = 1;
-        repeated int32 ids = 4;
-    }
-
-::
-
-    # myproto.options
-    MyMessage.name         max_size:40
-    MyMessage.ids          max_count:5
-
-The generator will automatically search for this file and read the
-options from it. The file format is as follows:
-
-* Lines starting with '#' or '//' are regarded as comments.
-* Blank lines are ignored.
-* All other lines should start with a field name pattern, followed by one or
-  more options. For example: *"MyMessage.myfield max_size:5 max_count:10"*.
-* The field name pattern is matched against a string of form *'Message.field'*.
-  For nested messages, the string is *'Message.SubMessage.field'*.
-* The field name pattern may use the notation recognized by Python fnmatch():
-
-  - *\** matches any part of string, like 'Message.\*' for all fields
-  - *\?* matches any single character
-  - *[seq]* matches any of characters 's', 'e' and 'q'
-  - *[!seq]* matches any other character
-
-* The options are written as *'option_name:option_value'* and several options
-  can be defined on same line, separated by whitespace.
-* Options defined later in the file override the ones specified earlier, so
-  it makes sense to define wildcard options first in the file and more specific
-  ones later.
-  
-If preferred, the name of the options file can be set using the command line
-switch *-f* to nanopb_generator.py.
-
-Defining the options on command line
-------------------------------------
-The nanopb_generator.py has a simple command line option *-s OPTION:VALUE*.
-The setting applies to the whole file that is being processed.
-
-Defining the options in the .proto file
----------------------------------------
-The .proto file format allows defining custom options for the fields.
-The nanopb library comes with *nanopb.proto* which does exactly that, allowing
-you do define the options directly in the .proto file::
-
-    import "nanopb.proto";
-    
-    message MyMessage {
-        required string name = 1 [(nanopb).max_size = 40];
-        repeated int32 ids = 4   [(nanopb).max_count = 5];
-    }
-
-A small complication is that you have to set the include path of protoc so that
-nanopb.proto can be found. This file, in turn, requires the file
-*google/protobuf/descriptor.proto*. This is usually installed under
-*/usr/include*. Therefore, to compile a .proto file which uses options, use a
-protoc command similar to::
-
-    protoc -I/usr/include -Inanopb/generator -I. -omessage.pb message.proto
-
-The options can be defined in file, message and field scopes::
-
-    option (nanopb_fileopt).max_size = 20; // File scope
-    message Message
-    {
-        option (nanopb_msgopt).max_size = 30; // Message scope
-        required string fieldsize = 1 [(nanopb).max_size = 40]; // Field scope
-    }
-
-
-
-
-
-
-
+============================  ================================================================================================
 
+The PB_MAX_REQUIRED_FIELDS, PB_FIELD_16BIT and PB_FIELD_32BIT settings allow raising some datatype limits to suit larger messages.
+Their need is recognized automatically by C-preprocessor #if-directives in the generated .pb.h files. The default setting is to use
+the smallest datatypes (least resources used).
 
 pb.h
 ====
@@ -304,76 +148,6 @@ Protocol Buffers wire types. These are used with `pb_encode_tag`_. ::
         PB_WT_32BIT  = 5
     } pb_wire_type_t;
 
-pb_extension_type_t
--------------------
-Defines the handler functions and auxiliary data for a field that extends
-another message. Usually autogenerated by *nanopb_generator.py*::
-
-    typedef struct {
-        bool (*decode)(pb_istream_t *stream, pb_extension_t *extension,
-                   uint32_t tag, pb_wire_type_t wire_type);
-        bool (*encode)(pb_ostream_t *stream, const pb_extension_t *extension);
-        const void *arg;
-    } pb_extension_type_t;
-
-In the normal case, the function pointers are *NULL* and the decoder and
-encoder use their internal implementations. The internal implementations
-assume that *arg* points to a *pb_field_t* that describes the field in question.
-
-To implement custom processing of unknown fields, you can provide pointers
-to your own functions. Their functionality is mostly the same as for normal
-callback fields, except that they get called for any unknown field when decoding.
-
-pb_extension_t
---------------
-Ties together the extension field type and the storage for the field value::
-
-    typedef struct {
-        const pb_extension_type_t *type;
-        void *dest;
-        pb_extension_t *next;
-    } pb_extension_t;
-
-:type:      Pointer to the structure that defines the callback functions.
-:dest:      Pointer to the variable that stores the field value
-            (as used by the default extension callback functions.)
-:next:      Pointer to the next extension handler, or *NULL*.
-
-PB_GET_ERROR
-------------
-Get the current error message from a stream, or a placeholder string if
-there is no error message::
-
-    #define PB_GET_ERROR(stream) (string expression)
-
-This should be used for printing errors, for example::
-
-    if (!pb_decode(...))
-    {
-        printf("Decode failed: %s\n", PB_GET_ERROR(stream));
-    }
-
-The macro only returns pointers to constant strings (in code memory),
-so that there is no need to release the returned pointer.
-
-PB_RETURN_ERROR
----------------
-Set the error message and return false::
-
-    #define PB_RETURN_ERROR(stream,msg) (sets error and returns false)
-
-This should be used to handle error conditions inside nanopb functions
-and user callback functions::
-
-    if (error_condition)
-    {
-        PB_RETURN_ERROR(stream, "something went wrong");
-    }
-
-The *msg* parameter must be a constant string.
-
-
-
 pb_encode.h
 ===========
 
@@ -523,17 +297,6 @@ In Protocol Buffers format, the submessage size must be written before the subme
 
 If the submessage contains callback fields, the callback function might misbehave and write out a different amount of data on the second call. This situation is recognized and *false* is returned, but garbage will be written to the output before the problem is detected.
 
-
-
-
-
-
-
-
-
-
-
-
 pb_decode.h
 ===========
 

example/fileproto.options

@@ -1,13 +0,0 @@
-# This file defines the nanopb-specific options for the messages defined
-# in fileproto.proto.
-#
-# If you come from high-level programming background, the hardcoded
-# maximum lengths may disgust you. However, if your microcontroller only
-# has a few kB of ram to begin with, setting reasonable limits for
-# filenames is ok.
-#
-# On the other hand, using the callback interface, it is not necessary
-# to set a limit on the number of files in the response.
-
-ListFilesRequest.path 	max_size:128
-FileInfo.name 		max_size:128

example/fileproto.proto

@@ -1,14 +1,22 @@
+import "nanopb.proto";
+
 // This defines protocol for a simple server that lists files.
 //
-// See also the nanopb-specific options in fileproto.options.
+// If you come from high-level programming background, the hardcoded
+// maximum lengths may disgust you. However, if your microcontroller only
+// has a few kB of ram to begin with, setting reasonable limits for
+// filenames is ok.
+//
+// On the other hand, using the callback interface, it is not necessary
+// to set a limit on the number of files in the response.
 
 message ListFilesRequest {
-    optional string path = 1 [default = "/"];
+    optional string path = 1 [default = "/", (nanopb).max_size = 128];
 }
 
 message FileInfo {
     required uint64 inode = 1;
-    required string name = 2;
+    required string name = 2 [(nanopb).max_size = 128];
 }
 
 message ListFilesResponse {

generator/Makefile

@@ -1,5 +1,2 @@
 nanopb_pb2.py: nanopb.proto
 	protoc --python_out=. -I /usr/include -I . nanopb.proto
-
-plugin_pb2.py: plugin.proto
-	protoc --python_out=. -I /usr/include -I . plugin.proto

generator/nanopb.proto

@@ -33,8 +33,6 @@ message NanoPBOptions {
   optional bool long_names = 4 [default = true];
   
   // Add 'packed' attribute to generated structs.
-  // Note: this cannot be used on CPUs that break on unaligned
-  // accesses to variables.
   optional bool packed_struct = 5 [default = false];
 }
 

generator/nanopb_generator.py

@@ -1,7 +1,5 @@
-#!/usr/bin/python
-
 '''Generate header file for nanopb from a ProtoBuf FileDescriptorSet.'''
-nanopb_version = "nanopb-0.2.3-dev"
+nanopb_version = "nanopb-0.2.1-dev"
 
 try:
     import google.protobuf.descriptor_pb2 as descriptor
@@ -237,23 +235,17 @@ class Field:
         else:
             return 'const %s %s_default%s = %s;' % (ctype, self.struct_name + self.name, array_decl, default)
     
-    def tags(self):
-        '''Return the #define for the tag number of this field.'''
-        identifier = '%s_%s_tag' % (self.struct_name, self.name)
-        return '#define %-40s %d\n' % (identifier, self.tag)
-    
     def pb_field_t(self, prev_field_name):
         '''Return the pb_field_t initializer to use in the constant array.
         prev_field_name is the name of the previous field or None.
         '''
-        result  = '    PB_FIELD2(%3d, ' % self.tag
+        result  = '    PB_FIELD(%3d, ' % self.tag
         result += '%-8s, ' % self.pbtype
         result += '%s, ' % self.rules
         result += '%s, ' % self.allocation
         result += '%s, ' % self.struct_name
         result += '%s, ' % self.name
         result += '%s, ' % (prev_field_name or self.name)
-        result += '%s, ' % ("first" if not prev_field_name else "other")
         
         if self.pbtype == 'MESSAGE':
             result += '&%s_fields)' % self.submsgname
@@ -278,72 +270,8 @@ class Field:
         return max(self.tag, self.max_size, self.max_count)        
 
 
-class ExtensionRange(Field):
-    def __init__(self, struct_name, range_start, field_options):
-        '''Implements a special pb_extension_t* field in an extensible message
-        structure. The range_start signifies the index at which the extensions
-        start. Not necessarily all tags above this are extensions, it is merely
-        a speed optimization.
-        '''
-        self.tag = range_start
-        self.struct_name = struct_name
-        self.name = 'extensions'
-        self.pbtype = 'EXTENSION'
-        self.rules = 'OPTIONAL'
-        self.allocation = 'CALLBACK'
-        self.ctype = 'pb_extension_t'
-        self.array_decl = ''
-        self.default = None
-        self.max_size = 0
-        self.max_count = 0
-        
-    def __str__(self):
-        return '    pb_extension_t *extensions;'
-    
-    def types(self):
-        return None
-    
-    def tags(self):
-        return ''
 
-class ExtensionField(Field):
-    def __init__(self, struct_name, desc, field_options):
-        self.fullname = struct_name + desc.name
-        self.extendee_name = names_from_type_name(desc.extendee)
-        Field.__init__(self, self.fullname + 'struct', desc, field_options)
-        
-        if self.rules != 'OPTIONAL':
-            self.skip = True
-        else:
-            self.skip = False
-            self.rules = 'OPTEXT'
 
-    def extension_decl(self):
-        '''Declaration of the extension type in the .pb.h file'''
-        if self.skip:
-            msg = '/* Extension field %s was skipped because only "optional"\n' % self.fullname
-            msg +='   type of extension fields is currently supported. */\n'
-            return msg
-        
-        return 'extern const pb_extension_type_t %s;\n' % self.fullname
-
-    def extension_def(self):
-        '''Definition of the extension type in the .pb.c file'''
-
-        if self.skip:
-            return ''
-
-        result  = 'typedef struct {\n'
-        result += str(self)
-        result += '\n} %s;\n\n' % self.struct_name
-        result += ('static const pb_field_t %s_field = \n  %s;\n\n' %
-                    (self.fullname, self.pb_field_t(None)))
-        result += 'const pb_extension_type_t %s = {\n' % self.fullname
-        result += '    NULL,\n'
-        result += '    NULL,\n'
-        result += '    &%s_field\n' % self.fullname
-        result += '};\n'
-        return result
 
 
 # ---------------------------------------------------------------------------
@@ -361,12 +289,6 @@ class Message:
             if field_options.type != nanopb_pb2.FT_IGNORE:
                 self.fields.append(Field(self.name, f, field_options))
         
-        if len(desc.extension_range) > 0:
-            field_options = get_nanopb_suboptions(desc, message_options, self.name + 'extensions')
-            range_start = min([r.start for r in desc.extension_range])
-            if field_options.type != nanopb_pb2.FT_IGNORE:
-                self.fields.append(ExtensionRange(self.name, range_start, field_options))
-        
         self.packed = message_options.packed_struct
         self.ordered_fields = self.fields[:]
         self.ordered_fields.sort()
@@ -431,6 +353,9 @@ class Message:
 
 
 
+
+
+
 # ---------------------------------------------------------------------------
 #                    Processing of entire .proto files
 # ---------------------------------------------------------------------------
@@ -450,23 +375,11 @@ def iterate_messages(desc, names = Names()):
         for x in iterate_messages(submsg, sub_names):
             yield x
 
-def iterate_extensions(desc, names = Names()):
-    '''Recursively find all extensions.
-    For each, yield name, FieldDescriptorProto.
-    '''
-    for extension in desc.extension:
-        yield names, extension
-
-    for subname, subdesc in iterate_messages(desc, names):
-        for extension in subdesc.extension:
-            yield subname, extension
-
 def parse_file(fdesc, file_options):
-    '''Takes a FileDescriptorProto and returns tuple (enums, messages, extensions).'''
+    '''Takes a FileDescriptorProto and returns tuple (enum, messages).'''
     
     enums = []
     messages = []
-    extensions = []
     
     if fdesc.package:
         base_name = Names(fdesc.package.split('.'))
@@ -484,11 +397,6 @@ def parse_file(fdesc, file_options):
             enum_options = get_nanopb_suboptions(enum, message_options, names + enum.name)
             enums.append(Enum(names, enum, enum_options))
     
-    for names, extension in iterate_extensions(fdesc, base_name):
-        field_options = get_nanopb_suboptions(extension, file_options, names)
-        if field_options.type != nanopb_pb2.FT_IGNORE:
-            extensions.append(ExtensionField(names, extension, field_options))
-    
     # Fix field default values where enum short names are used.
     for enum in enums:
         if not enum.options.long_names:
@@ -498,7 +406,7 @@ def parse_file(fdesc, file_options):
                         idx = enum.value_longnames.index(field.default)
                         field.default = enum.values[idx][0]
     
-    return enums, messages, extensions
+    return enums, messages
 
 def toposort2(data):
     '''Topological sort.
@@ -541,7 +449,7 @@ def make_identifier(headername):
             result += '_'
     return result
 
-def generate_header(dependencies, headername, enums, messages, extensions, options):
+def generate_header(dependencies, headername, enums, messages, options):
     '''Generate content for a header file.
     Generates strings, which should be concatenated and stored to file.
     '''
@@ -576,24 +484,12 @@ def generate_header(dependencies, headername, enums, messages, extensions, optio
     for msg in sort_dependencies(messages):
         yield msg.types()
         yield str(msg) + '\n\n'
-    
-    if extensions:
-        yield '/* Extensions */\n'
-        for extension in extensions:
-            yield extension.extension_decl()
-        yield '\n'
         
     yield '/* Default values for struct fields */\n'
     for msg in messages:
         yield msg.default_decl(True)
     yield '\n'
     
-    yield '/* Field tags (for use in manual encoding/decoding) */\n'
-    for msg in sort_dependencies(messages):
-        for field in msg.fields:
-            yield field.tags()
-    yield '\n'
-    
     yield '/* Struct field encoding specification for nanopb */\n'
     for msg in messages:
         yield msg.fields_declaration() + '\n'
@@ -605,7 +501,7 @@ def generate_header(dependencies, headername, enums, messages, extensions, optio
     # End of header
     yield '\n#endif\n'
 
-def generate_source(headername, enums, messages, extensions, options):
+def generate_source(headername, enums, messages):
     '''Generate content for a source file.'''
     
     yield '/* Automatically generated nanopb constant definitions */\n'
@@ -620,11 +516,7 @@ def generate_source(headername, enums, messages, extensions, options):
     
     for msg in messages:
         yield msg.fields_definition() + '\n\n'
-    
-    for ext in extensions:
-        yield ext.extension_def() + '\n'
         
-    # Add checks for numeric limits
     if messages:
         count_required_fields = lambda m: len([f for f in msg.fields if f.rules == 'REQUIRED'])
         largest_msg = max(messages, key = count_required_fields)
@@ -636,6 +528,7 @@ def generate_source(headername, enums, messages, extensions, options):
             yield '       setting PB_MAX_REQUIRED_FIELDS to %d or more.\n' % largest_count
             yield '#endif\n'
     
+    # Add checks for numeric limits
     worst = 0
     worst_field = ''
     checks = []
@@ -783,126 +676,72 @@ optparser.add_option("-v", "--verbose", dest="verbose", action="store_true", def
 optparser.add_option("-s", dest="settings", metavar="OPTION:VALUE", action="append", default=[],
     help="Set generator option (max_size, max_count etc.).")
 
-def process_file(filename, fdesc, options):
-    '''Process a single file.
-    filename: The full path to the .proto or .pb source file, as string.
-    fdesc: The loaded FileDescriptorSet, or None to read from the input file.
-    options: Command line options as they come from OptionsParser.
-    
-    Returns a dict:
-        {'headername': Name of header file,
-         'headerdata': Data for the .h header file,
-         'sourcename': Name of the source code file,
-         'sourcedata': Data for the .c source code file
-        }
-    '''
-    toplevel_options = nanopb_pb2.NanoPBOptions()
-    for s in options.settings:
-        text_format.Merge(s, toplevel_options)
-    
-    if not fdesc:
-        data = open(filename, 'rb').read()
-        fdesc = descriptor.FileDescriptorSet.FromString(data).file[0]
-    
-    # Check if there is a separate .options file
-    try:
-        optfilename = options.options_file % os.path.splitext(filename)[0]
-    except TypeError:
-        # No %s specified, use the filename as-is
-        optfilename = options.options_file
-    
-    if options.verbose:
-        print 'Reading options from ' + optfilename
-    
-    if os.path.isfile(optfilename):
-        Globals.separate_options = read_options_file(open(optfilename, "rU"))
-    else:
-        Globals.separate_options = []
-    
-    # Parse the file
-    file_options = get_nanopb_suboptions(fdesc, toplevel_options, Names([filename]))
-    enums, messages, extensions = parse_file(fdesc, file_options)
-    
-    # Decide the file names
-    noext = os.path.splitext(filename)[0]
-    headername = noext + '.' + options.extension + '.h'
-    sourcename = noext + '.' + options.extension + '.c'
-    headerbasename = os.path.basename(headername)
-    
-    # List of .proto files that should not be included in the C header file
-    # even if they are mentioned in the source .proto.
-    excludes = ['nanopb.proto', 'google/protobuf/descriptor.proto'] + options.exclude
-    dependencies = [d for d in fdesc.dependency if d not in excludes]
-    
-    headerdata = ''.join(generate_header(dependencies, headerbasename, enums,
-                                         messages, extensions, options))
-
-    sourcedata = ''.join(generate_source(headerbasename, enums,
-                                         messages, extensions, options))
-
-    return {'headername': headername, 'headerdata': headerdata,
-            'sourcename': sourcename, 'sourcedata': sourcedata}
-    
-def main_cli():
-    '''Main function when invoked directly from the command line.'''
-    
-    options, filenames = optparser.parse_args()
+def process(filenames, options):
+    '''Process the files given on the command line.'''
     
     if not filenames:
         optparser.print_help()
-        sys.exit(1)
+        return False
     
     if options.quiet:
         options.verbose = False
 
     Globals.verbose_options = options.verbose
     
+    toplevel_options = nanopb_pb2.NanoPBOptions()
+    for s in options.settings:
+        text_format.Merge(s, toplevel_options)
+    
     for filename in filenames:
-        results = process_file(filename, None, options)
+        data = open(filename, 'rb').read()
+        fdesc = descriptor.FileDescriptorSet.FromString(data)
+        
+        # Check if any separate options are specified
+        try:
+            optfilename = options.options_file % os.path.splitext(filename)[0]
+        except TypeError:
+            # No %s specified, use the filename as-is
+            optfilename = options.options_file
+        
+        if options.verbose:
+            print 'Reading options from ' + optfilename
+        
+        if os.path.isfile(optfilename):
+            Globals.separate_options = read_options_file(open(optfilename, "rU"))
+        else:
+            Globals.separate_options = []
+        
+        # Parse the file
+        file_options = get_nanopb_suboptions(fdesc.file[0], toplevel_options, Names([filename]))
+        enums, messages = parse_file(fdesc.file[0], file_options)
+        
+        noext = os.path.splitext(filename)[0]
+        headername = noext + '.' + options.extension + '.h'
+        sourcename = noext + '.' + options.extension + '.c'
+        headerbasename = os.path.basename(headername)
         
         if not options.quiet:
-            print "Writing to " + results['headername'] + " and " + results['sourcename']
-    
-        open(results['headername'], 'w').write(results['headerdata'])
-        open(results['sourcename'], 'w').write(results['sourcedata'])        
+            print "Writing to " + headername + " and " + sourcename
+        
+        # List of .proto files that should not be included in the C header file
+        # even if they are mentioned in the source .proto.
+        excludes = ['nanopb.proto', 'google/protobuf/descriptor.proto'] + options.exclude
+        dependencies = [d for d in fdesc.file[0].dependency if d not in excludes]
+        
+        header = open(headername, 'w')
+        for part in generate_header(dependencies, headerbasename, enums, messages, options):
+            header.write(part)
 
-def main_plugin():
-    '''Main function when invoked as a protoc plugin.'''
+        source = open(sourcename, 'w')
+        for part in generate_source(headerbasename, enums, messages):
+            source.write(part)
 
-    import plugin_pb2
-    data = sys.stdin.read()
-    request = plugin_pb2.CodeGeneratorRequest.FromString(data)
-    
-    import shlex
-    args = shlex.split(request.parameter)
-    options, dummy = optparser.parse_args(args)
-    
-    # We can't go printing stuff to stdout
-    Globals.verbose_options = False
-    options.verbose = False
-    options.quiet = True
-    
-    response = plugin_pb2.CodeGeneratorResponse()
-    
-    for filename in request.file_to_generate:
-        for fdesc in request.proto_file:
-            if fdesc.name == filename:
-                results = process_file(filename, fdesc, options)
-                
-                f = response.file.add()
-                f.name = results['headername']
-                f.content = results['headerdata']
-
-                f = response.file.add()
-                f.name = results['sourcename']
-                f.content = results['sourcedata']    
-    
-    sys.stdout.write(response.SerializeToString())
+    return True
 
 if __name__ == '__main__':
-    # Check if we are running as a plugin under protoc
-    if 'protoc-gen-' in sys.argv[0]:
-        main_plugin()
-    else:
-        main_cli()
-
+    options, filenames = optparser.parse_args()
+    status = process(filenames, options)
+    
+    if not status:
+        sys.exit(1)
+    

generator/plugin.proto

@@ -1,145 +0,0 @@
-// Protocol Buffers - Google's data interchange format
-// Copyright 2008 Google Inc.  All rights reserved.
-// http://code.google.com/p/protobuf/
-//
-// Redistribution and use in source and binary forms, with or without
-// modification, are permitted provided that the following conditions are
-// met:
-//
-//     * Redistributions of source code must retain the above copyright
-// notice, this list of conditions and the following disclaimer.
-//     * Redistributions in binary form must reproduce the above
-// copyright notice, this list of conditions and the following disclaimer
-// in the documentation and/or other materials provided with the
-// distribution.
-//     * Neither the name of Google Inc. nor the names of its
-// contributors may be used to endorse or promote products derived from
-// this software without specific prior written permission.
-//
-// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-// Author: kenton@google.com (Kenton Varda)
-//
-// WARNING:  The plugin interface is currently EXPERIMENTAL and is subject to
-//   change.
-//
-// protoc (aka the Protocol Compiler) can be extended via plugins.  A plugin is
-// just a program that reads a CodeGeneratorRequest from stdin and writes a
-// CodeGeneratorResponse to stdout.
-//
-// Plugins written using C++ can use google/protobuf/compiler/plugin.h instead
-// of dealing with the raw protocol defined here.
-//
-// A plugin executable needs only to be placed somewhere in the path.  The
-// plugin should be named "protoc-gen-$NAME", and will then be used when the
-// flag "--${NAME}_out" is passed to protoc.
-
-package google.protobuf.compiler;
-
-import "google/protobuf/descriptor.proto";
-
-// An encoded CodeGeneratorRequest is written to the plugin's stdin.
-message CodeGeneratorRequest {
-  // The .proto files that were explicitly listed on the command-line.  The
-  // code generator should generate code only for these files.  Each file's
-  // descriptor will be included in proto_file, below.
-  repeated string file_to_generate = 1;
-
-  // The generator parameter passed on the command-line.
-  optional string parameter = 2;
-
-  // FileDescriptorProtos for all files in files_to_generate and everything
-  // they import.  The files will appear in topological order, so each file
-  // appears before any file that imports it.
-  //
-  // protoc guarantees that all proto_files will be written after
-  // the fields above, even though this is not technically guaranteed by the
-  // protobuf wire format.  This theoretically could allow a plugin to stream
-  // in the FileDescriptorProtos and handle them one by one rather than read
-  // the entire set into memory at once.  However, as of this writing, this
-  // is not similarly optimized on protoc's end -- it will store all fields in
-  // memory at once before sending them to the plugin.
-  repeated FileDescriptorProto proto_file = 15;
-}
-
-// The plugin writes an encoded CodeGeneratorResponse to stdout.
-message CodeGeneratorResponse {
-  // Error message.  If non-empty, code generation failed.  The plugin process
-  // should exit with status code zero even if it reports an error in this way.
-  //
-  // This should be used to indicate errors in .proto files which prevent the
-  // code generator from generating correct code.  Errors which indicate a
-  // problem in protoc itself -- such as the input CodeGeneratorRequest being
-  // unparseable -- should be reported by writing a message to stderr and
-  // exiting with a non-zero status code.
-  optional string error = 1;
-
-  // Represents a single generated file.
-  message File {
-    // The file name, relative to the output directory.  The name must not
-    // contain "." or ".." components and must be relative, not be absolute (so,
-    // the file cannot lie outside the output directory).  "/" must be used as
-    // the path separator, not "\".
-    //
-    // If the name is omitted, the content will be appended to the previous
-    // file.  This allows the generator to break large files into small chunks,
-    // and allows the generated text to be streamed back to protoc so that large
-    // files need not reside completely in memory at one time.  Note that as of
-    // this writing protoc does not optimize for this -- it will read the entire
-    // CodeGeneratorResponse before writing files to disk.
-    optional string name = 1;
-
-    // If non-empty, indicates that the named file should already exist, and the
-    // content here is to be inserted into that file at a defined insertion
-    // point.  This feature allows a code generator to extend the output
-    // produced by another code generator.  The original generator may provide
-    // insertion points by placing special annotations in the file that look
-    // like:
-    //   @@protoc_insertion_point(NAME)
-    // The annotation can have arbitrary text before and after it on the line,
-    // which allows it to be placed in a comment.  NAME should be replaced with
-    // an identifier naming the point -- this is what other generators will use
-    // as the insertion_point.  Code inserted at this point will be placed
-    // immediately above the line containing the insertion point (thus multiple
-    // insertions to the same point will come out in the order they were added).
-    // The double-@ is intended to make it unlikely that the generated code
-    // could contain things that look like insertion points by accident.
-    //
-    // For example, the C++ code generator places the following line in the
-    // .pb.h files that it generates:
-    //   // @@protoc_insertion_point(namespace_scope)
-    // This line appears within the scope of the file's package namespace, but
-    // outside of any particular class.  Another plugin can then specify the
-    // insertion_point "namespace_scope" to generate additional classes or
-    // other declarations that should be placed in this scope.
-    //
-    // Note that if the line containing the insertion point begins with
-    // whitespace, the same whitespace will be added to every line of the
-    // inserted text.  This is useful for languages like Python, where
-    // indentation matters.  In these languages, the insertion point comment
-    // should be indented the same amount as any inserted code will need to be
-    // in order to work correctly in that context.
-    //
-    // The code generator that generates the initial file and the one which
-    // inserts into it must both run as part of a single invocation of protoc.
-    // Code generators are executed in the order in which they appear on the
-    // command line.
-    //
-    // If |insertion_point| is present, |name| must also be present.
-    optional string insertion_point = 2;
-
-    // The file contents.
-    optional string content = 15;
-  }
-  repeated File file = 15;
-}

generator/plugin_pb2.py

@@ -1,161 +0,0 @@
-# Generated by the protocol buffer compiler.  DO NOT EDIT!
-
-from google.protobuf import descriptor
-from google.protobuf import message
-from google.protobuf import reflection
-from google.protobuf import descriptor_pb2
-# @@protoc_insertion_point(imports)
-
-
-import google.protobuf.descriptor_pb2
-
-DESCRIPTOR = descriptor.FileDescriptor(
-  name='plugin.proto',
-  package='google.protobuf.compiler',
-  serialized_pb='\n\x0cplugin.proto\x12\x18google.protobuf.compiler\x1a google/protobuf/descriptor.proto\"}\n\x14\x43odeGeneratorRequest\x12\x18\n\x10\x66ile_to_generate\x18\x01 \x03(\t\x12\x11\n\tparameter\x18\x02 \x01(\t\x12\x38\n\nproto_file\x18\x0f \x03(\x0b\x32$.google.protobuf.FileDescriptorProto\"\xaa\x01\n\x15\x43odeGeneratorResponse\x12\r\n\x05\x65rror\x18\x01 \x01(\t\x12\x42\n\x04\x66ile\x18\x0f \x03(\x0b\x32\x34.google.protobuf.compiler.CodeGeneratorResponse.File\x1a>\n\x04\x46ile\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x17\n\x0finsertion_point\x18\x02 \x01(\t\x12\x0f\n\x07\x63ontent\x18\x0f \x01(\t')
-
-
-
-
-_CODEGENERATORREQUEST = descriptor.Descriptor(
-  name='CodeGeneratorRequest',
-  full_name='google.protobuf.compiler.CodeGeneratorRequest',
-  filename=None,
-  file=DESCRIPTOR,
-  containing_type=None,
-  fields=[
-    descriptor.FieldDescriptor(
-      name='file_to_generate', full_name='google.protobuf.compiler.CodeGeneratorRequest.file_to_generate', index=0,
-      number=1, type=9, cpp_type=9, label=3,
-      has_default_value=False, default_value=[],
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='parameter', full_name='google.protobuf.compiler.CodeGeneratorRequest.parameter', index=1,
-      number=2, type=9, cpp_type=9, label=1,
-      has_default_value=False, default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='proto_file', full_name='google.protobuf.compiler.CodeGeneratorRequest.proto_file', index=2,
-      number=15, type=11, cpp_type=10, label=3,
-      has_default_value=False, default_value=[],
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-  ],
-  extensions=[
-  ],
-  nested_types=[],
-  enum_types=[
-  ],
-  options=None,
-  is_extendable=False,
-  extension_ranges=[],
-  serialized_start=76,
-  serialized_end=201,
-)
-
-
-_CODEGENERATORRESPONSE_FILE = descriptor.Descriptor(
-  name='File',
-  full_name='google.protobuf.compiler.CodeGeneratorResponse.File',
-  filename=None,
-  file=DESCRIPTOR,
-  containing_type=None,
-  fields=[
-    descriptor.FieldDescriptor(
-      name='name', full_name='google.protobuf.compiler.CodeGeneratorResponse.File.name', index=0,
-      number=1, type=9, cpp_type=9, label=1,
-      has_default_value=False, default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='insertion_point', full_name='google.protobuf.compiler.CodeGeneratorResponse.File.insertion_point', index=1,
-      number=2, type=9, cpp_type=9, label=1,
-      has_default_value=False, default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='content', full_name='google.protobuf.compiler.CodeGeneratorResponse.File.content', index=2,
-      number=15, type=9, cpp_type=9, label=1,
-      has_default_value=False, default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-  ],
-  extensions=[
-  ],
-  nested_types=[],
-  enum_types=[
-  ],
-  options=None,
-  is_extendable=False,
-  extension_ranges=[],
-  serialized_start=312,
-  serialized_end=374,
-)
-
-_CODEGENERATORRESPONSE = descriptor.Descriptor(
-  name='CodeGeneratorResponse',
-  full_name='google.protobuf.compiler.CodeGeneratorResponse',
-  filename=None,
-  file=DESCRIPTOR,
-  containing_type=None,
-  fields=[
-    descriptor.FieldDescriptor(
-      name='error', full_name='google.protobuf.compiler.CodeGeneratorResponse.error', index=0,
-      number=1, type=9, cpp_type=9, label=1,
-      has_default_value=False, default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='file', full_name='google.protobuf.compiler.CodeGeneratorResponse.file', index=1,
-      number=15, type=11, cpp_type=10, label=3,
-      has_default_value=False, default_value=[],
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-  ],
-  extensions=[
-  ],
-  nested_types=[_CODEGENERATORRESPONSE_FILE, ],
-  enum_types=[
-  ],
-  options=None,
-  is_extendable=False,
-  extension_ranges=[],
-  serialized_start=204,
-  serialized_end=374,
-)
-
-_CODEGENERATORREQUEST.fields_by_name['proto_file'].message_type = google.protobuf.descriptor_pb2._FILEDESCRIPTORPROTO
-_CODEGENERATORRESPONSE_FILE.containing_type = _CODEGENERATORRESPONSE;
-_CODEGENERATORRESPONSE.fields_by_name['file'].message_type = _CODEGENERATORRESPONSE_FILE
-DESCRIPTOR.message_types_by_name['CodeGeneratorRequest'] = _CODEGENERATORREQUEST
-DESCRIPTOR.message_types_by_name['CodeGeneratorResponse'] = _CODEGENERATORRESPONSE
-
-class CodeGeneratorRequest(message.Message):
-  __metaclass__ = reflection.GeneratedProtocolMessageType
-  DESCRIPTOR = _CODEGENERATORREQUEST
-  
-  # @@protoc_insertion_point(class_scope:google.protobuf.compiler.CodeGeneratorRequest)
-
-class CodeGeneratorResponse(message.Message):
-  __metaclass__ = reflection.GeneratedProtocolMessageType
-  
-  class File(message.Message):
-    __metaclass__ = reflection.GeneratedProtocolMessageType
-    DESCRIPTOR = _CODEGENERATORRESPONSE_FILE
-    
-    # @@protoc_insertion_point(class_scope:google.protobuf.compiler.CodeGeneratorResponse.File)
-  DESCRIPTOR = _CODEGENERATORRESPONSE
-  
-  # @@protoc_insertion_point(class_scope:google.protobuf.compiler.CodeGeneratorResponse)
-
-# @@protoc_insertion_point(module_scope)

pb.h

@@ -1,61 +1,13 @@
-/* Common parts of the nanopb library. Most of these are quite low-level
- * stuff. For the high-level interface, see pb_encode.h and pb_decode.h.
- */
-
 #ifndef _PB_H_
 #define _PB_H_
 
-/*****************************************************************
- * Nanopb compilation time options. You can change these here by *
- * uncommenting the lines, or on the compiler command line.      *
- *****************************************************************/
-
-/* Define this if your CPU architecture is big endian, i.e. it
- * stores the most-significant byte first. */
-/* #define __BIG_ENDIAN__ 1 */
-
-/* Increase the number of required fields that are tracked.
- * A compiler warning will tell if you need this. */
-/* #define PB_MAX_REQUIRED_FIELDS 256 */
-
-/* Add support for tag numbers > 255 and fields larger than 255 bytes. */
-/* #define PB_FIELD_16BIT 1 */
-
-/* Add support for tag numbers > 65536 and fields larger than 65536 bytes. */
-/* #define PB_FIELD_32BIT 1 */
-
-/* Disable support for error messages in order to save some code space. */
-/* #define PB_NO_ERRMSG 1 */
-
-/* Disable support for custom streams (support only memory buffers). */
-/* #define PB_BUFFER_ONLY 1 */
-
-/* Switch back to the old-style callback function signature.
- * This was the default until nanopb-0.2.1. */
-/* #define PB_OLD_CALLBACK_STYLE */
-
-
-/******************************************************************
- * You usually don't need to change anything below this line.     *
- * Feel free to look around and use the defined macros, though.   *
- ******************************************************************/
-
-
-/* Version of the nanopb library. Just in case you want to check it in
- * your own program. */
-#define NANOPB_VERSION nanopb-0.2.3-dev
-
-/* Include all the system headers needed by nanopb. You will need the
- * definitions of the following:
- * - strlen, memcpy, memset functions
- * - [u]int8_t, [u]int16_t, [u]int32_t, [u]int64_t
- * - size_t
- * - bool
- *
- * If you don't have the standard header files, you can instead provide
- * a custom header that defines or includes all this. In that case,
- * define PB_SYSTEM_HEADER to the path of this file.
+/* pb.h: Common parts for nanopb library.
+ * Most of these are quite low-level stuff. For the high-level interface,
+ * see pb_encode.h or pb_decode.h
  */
+
+#define NANOPB_VERSION nanopb-0.2.1-dev
+
 #ifdef PB_SYSTEM_HEADER
 #include PB_SYSTEM_HEADER
 #else
@@ -78,7 +30,7 @@
 #   define PB_PACKED_STRUCT_START _Pragma("pack(push, 1)")
 #   define PB_PACKED_STRUCT_END _Pragma("pack(pop)")
 #   define pb_packed
-#elif defined(_MSC_VER) && (_MSC_VER >= 1500)
+#elif defined(_MSC_VER)
     /* For Microsoft Visual C++ */
 #   define PB_PACKED_STRUCT_START __pragma(pack(push, 1))
 #   define PB_PACKED_STRUCT_END __pragma(pack(pop))
@@ -90,7 +42,7 @@
 #   define pb_packed
 #endif
 
-/* Handly macro for suppressing unreferenced-parameter compiler warnings. */
+/* Handly macro for suppressing unreferenced-parameter compiler warnings.    */
 #ifndef UNUSED
 #define UNUSED(x) (void)(x)
 #endif
@@ -104,7 +56,8 @@
 #define STATIC_ASSERT_MSG_(MSG, LINE, COUNTER) static_assertion_##MSG##LINE##COUNTER
 #endif
 
-/* Number of required fields to keep track of. */
+/* Number of required fields to keep track of
+ * (change here or on compiler command line). */
 #ifndef PB_MAX_REQUIRED_FIELDS
 #define PB_MAX_REQUIRED_FIELDS 64
 #endif
@@ -125,7 +78,9 @@
 
 typedef uint8_t pb_type_t;
 
-/**** Field data types ****/
+/************************
+ * Field contents types *
+ ************************/
 
 /* Numeric types */
 #define PB_LTYPE_VARINT 0x00 /* int32, uint32, int64, uint64, bool, enum */
@@ -148,22 +103,22 @@ typedef uint8_t pb_type_t;
  * submsg_fields is pointer to field descriptions */
 #define PB_LTYPE_SUBMESSAGE 0x06
 
-/* Extension pseudo-field
- * The field contains a pointer to pb_extension_t */
-#define PB_LTYPE_EXTENSION 0x07
-
 /* Number of declared LTYPES */
-#define PB_LTYPES_COUNT 8
+#define PB_LTYPES_COUNT 7
 #define PB_LTYPE_MASK 0x0F
 
-/**** Field repetition rules ****/
+/**************************
+ * Field repetition rules *
+ **************************/
 
 #define PB_HTYPE_REQUIRED 0x00
 #define PB_HTYPE_OPTIONAL 0x10
 #define PB_HTYPE_REPEATED 0x20
 #define PB_HTYPE_MASK     0x30
 
-/**** Field allocation types ****/
+/********************
+ * Allocation types *
+ ********************/
  
 #define PB_ATYPE_STATIC   0x00
 #define PB_ATYPE_CALLBACK 0x40
@@ -274,114 +229,50 @@ typedef enum {
     PB_WT_32BIT  = 5
 } pb_wire_type_t;
 
-/* Structure for defining the handling of unknown/extension fields.
- * Usually the pb_extension_type_t structure is automatically generated,
- * while the pb_extension_t structure is created by the user. However,
- * if you want to catch all unknown fields, you can also create a custom
- * pb_extension_type_t with your own callback.
- */
-typedef struct _pb_extension_type_t pb_extension_type_t;
-typedef struct _pb_extension_t pb_extension_t;
-struct _pb_extension_type_t {
-    /* Called for each unknown field in the message.
-     * If you handle the field, read off all of its data and return true.
-     * If you do not handle the field, do not read anything and return true.
-     * If you run into an error, return false.
-     * Set to NULL for default handler.
-     */
-    bool (*decode)(pb_istream_t *stream, pb_extension_t *extension,
-                   uint32_t tag, pb_wire_type_t wire_type);
-    
-    /* Called once after all regular fields have been encoded.
-     * If you have something to write, do so and return true.
-     * If you do not have anything to write, just return true.
-     * If you run into an error, return false.
-     * Set to NULL for default handler.
-     */
-    bool (*encode)(pb_ostream_t *stream, const pb_extension_t *extension);
-    
-    /* Free field for use by the callback. */
-    const void *arg;
-};
-
-struct _pb_extension_t {
-    /* Type describing the extension field. Usually you'll initialize
-     * this to a pointer to the automatically generated structure. */
-    const pb_extension_type_t *type;
-    
-    /* Destination for the decoded data. This must match the datatype
-     * of the extension field. */
-    void *dest;
-    
-    /* Pointer to the next extension handler, or NULL.
-     * If this extension does not match a field, the next handler is
-     * automatically called. */
-    pb_extension_t *next;
-};
-
 /* These macros are used to declare pb_field_t's in the constant array. */
-/* Size of a structure member, in bytes. */
 #define pb_membersize(st, m) (sizeof ((st*)0)->m)
-/* Number of entries in an array. */
 #define pb_arraysize(st, m) (pb_membersize(st, m) / pb_membersize(st, m[0]))
-/* Delta from start of one member to the start of another member. */
 #define pb_delta(st, m1, m2) ((int)offsetof(st, m1) - (int)offsetof(st, m2))
-/* Delta from start of structure to member. */
-#define pb_fielddelta_first(st, m1, m2) (offsetof(st, m1))
-/* Delta from end of one field to start of another field. */
-#define pb_fielddelta_other(st, m1, m2) (offsetof(st, m1) - offsetof(st, m2) - pb_membersize(st, m2))
-/* Choose between pb_fielddelta_first and pb_fielddelta_other (backwards compatibility) */
-#define pb_fielddelta_choose(st, m1, m2) (int)(offsetof(st, m1) == offsetof(st, m2) \
-                                  ? pb_fielddelta_first(st, m1, m2) \
-                                  : pb_fielddelta_other(st, m1, m2))
+#define pb_delta_end(st, m1, m2) (int)(offsetof(st, m1) == offsetof(st, m2) \
+                                  ? offsetof(st, m1) \
+                                  : offsetof(st, m1) - offsetof(st, m2) - pb_membersize(st, m2))
 #define PB_LAST_FIELD {0,(pb_type_t) 0,0,0,0,0,0}
 
 /* Required fields are the simplest. They just have delta (padding) from
  * previous field end, and the size of the field. Pointer is used for
  * submessages and default values.
  */
-#define PB_REQUIRED_STATIC(tag, st, m, fd, ltype, ptr) \
+#define PB_REQUIRED_STATIC(tag, st, m, pm, ltype, ptr) \
     {tag, PB_ATYPE_STATIC | PB_HTYPE_REQUIRED | ltype, \
-    fd, 0, pb_membersize(st, m), 0, ptr}
+    pb_delta_end(st, m, pm), 0, pb_membersize(st, m), 0, ptr}
 
 /* Optional fields add the delta to the has_ variable. */
-#define PB_OPTIONAL_STATIC(tag, st, m, fd, ltype, ptr) \
+#define PB_OPTIONAL_STATIC(tag, st, m, pm, ltype, ptr) \
     {tag, PB_ATYPE_STATIC | PB_HTYPE_OPTIONAL | ltype, \
-    fd, \
+    pb_delta_end(st, m, pm), \
     pb_delta(st, has_ ## m, m), \
     pb_membersize(st, m), 0, ptr}
 
 /* Repeated fields have a _count field and also the maximum number of entries. */
-#define PB_REPEATED_STATIC(tag, st, m, fd, ltype, ptr) \
+#define PB_REPEATED_STATIC(tag, st, m, pm, ltype, ptr) \
     {tag, PB_ATYPE_STATIC | PB_HTYPE_REPEATED | ltype, \
-    fd, \
+    pb_delta_end(st, m, pm), \
     pb_delta(st, m ## _count, m), \
     pb_membersize(st, m[0]), \
     pb_arraysize(st, m), ptr}
 
 /* Callbacks are much like required fields except with special datatype. */
-#define PB_REQUIRED_CALLBACK(tag, st, m, fd, ltype, ptr) \
+#define PB_REQUIRED_CALLBACK(tag, st, m, pm, ltype, ptr) \
     {tag, PB_ATYPE_CALLBACK | PB_HTYPE_REQUIRED | ltype, \
-    fd, 0, pb_membersize(st, m), 0, ptr}
+    pb_delta_end(st, m, pm), 0, pb_membersize(st, m), 0, ptr}
 
-#define PB_OPTIONAL_CALLBACK(tag, st, m, fd, ltype, ptr) \
+#define PB_OPTIONAL_CALLBACK(tag, st, m, pm, ltype, ptr) \
     {tag, PB_ATYPE_CALLBACK | PB_HTYPE_OPTIONAL | ltype, \
-    fd, 0, pb_membersize(st, m), 0, ptr}
+    pb_delta_end(st, m, pm), 0, pb_membersize(st, m), 0, ptr}
     
-#define PB_REPEATED_CALLBACK(tag, st, m, fd, ltype, ptr) \
+#define PB_REPEATED_CALLBACK(tag, st, m, pm, ltype, ptr) \
     {tag, PB_ATYPE_CALLBACK | PB_HTYPE_REPEATED | ltype, \
-    fd, 0, pb_membersize(st, m), 0, ptr}
-
-/* Optional extensions don't have the has_ field, as that would be redundant. */
-#define PB_OPTEXT_STATIC(tag, st, m, fd, ltype, ptr) \
-    {tag, PB_ATYPE_STATIC | PB_HTYPE_OPTIONAL | ltype, \
-    0, \
-    0, \
-    pb_membersize(st, m), 0, ptr}
-
-#define PB_OPTEXT_CALLBACK(tag, st, m, fd, ltype, ptr) \
-    {tag, PB_ATYPE_CALLBACK | PB_HTYPE_OPTIONAL | ltype, \
-    0, 0, pb_membersize(st, m), 0, ptr}
+    pb_delta_end(st, m, pm), 0, pb_membersize(st, m), 0, ptr}
 
 /* The mapping from protobuf types to LTYPEs is done using these macros. */
 #define PB_LTYPE_MAP_BOOL       PB_LTYPE_VARINT
@@ -401,14 +292,13 @@ struct _pb_extension_t {
 #define PB_LTYPE_MAP_STRING     PB_LTYPE_STRING
 #define PB_LTYPE_MAP_UINT32     PB_LTYPE_VARINT
 #define PB_LTYPE_MAP_UINT64     PB_LTYPE_VARINT
-#define PB_LTYPE_MAP_EXTENSION  PB_LTYPE_EXTENSION
 
 /* This is the actual macro used in field descriptions.
  * It takes these arguments:
  * - Field tag number
  * - Field type:   BOOL, BYTES, DOUBLE, ENUM, FIXED32, FIXED64,
  *                 FLOAT, INT32, INT64, MESSAGE, SFIXED32, SFIXED64
- *                 SINT32, SINT64, STRING, UINT32, UINT64 or EXTENSION
+ *                 SINT32, SINT64, STRING, UINT32 or UINT64
  * - Field rules:  REQUIRED, OPTIONAL or REPEATED
  * - Allocation:   STATIC or CALLBACK
  * - Message name
@@ -418,19 +308,8 @@ struct _pb_extension_t {
  */
 
 #define PB_FIELD(tag, type, rules, allocation, message, field, prevfield, ptr) \
-    PB_ ## rules ## _ ## allocation(tag, message, field, \
-        pb_fielddelta_choose(message, field, prevfield), \
-        PB_LTYPE_MAP_ ## type, ptr)
-
-/* This is a new version of the macro used by nanopb generator from
- * version 0.2.3 onwards. It avoids the use of a ternary expression in
- * the initialization, which confused some compilers.
- */
-#define PB_FIELD2(tag, type, rules, allocation, message, field, prevfield, pos, ptr) \
-    PB_ ## rules ## _ ## allocation(tag, message, field, \
-        pb_fielddelta_ ## pos(message, field, prevfield), \
-        PB_LTYPE_MAP_ ## type, ptr)
-
+    PB_ ## rules ## _ ## allocation(tag, message, field, prevfield, \
+                                    PB_LTYPE_MAP_ ## type, ptr)
 
 /* These macros are used for giving out error messages.
  * They are mostly a debugging aid; the main error information

pb_decode.c

@@ -28,8 +28,7 @@ static const pb_decoder_t PB_DECODERS[PB_LTYPES_COUNT] = {
     
     &pb_dec_bytes,
     &pb_dec_string,
-    &pb_dec_submessage,
-    NULL /* extensions */
+    &pb_dec_submessage
 };
 
 /**************
@@ -309,12 +308,12 @@ static bool pb_field_next(pb_field_iterator_t *iter)
         prev_size *= iter->pos->array_size;
     }
     
-    if (iter->pos->tag == 0)
-        return false; /* Only happens with empty message types */
-    
     if (PB_HTYPE(iter->pos->type) == PB_HTYPE_REQUIRED)
         iter->required_field_index++;
     
+    if (iter->pos->tag == 0)
+        return false; /* Only happens with empty message types */
+    
     iter->pos++;
     iter->field_index++;
     if (iter->pos->tag == 0)
@@ -332,22 +331,6 @@ static bool pb_field_next(pb_field_iterator_t *iter)
     return notwrapped;
 }
 
-static bool checkreturn pb_field_find(pb_field_iterator_t *iter, uint32_t tag)
-{
-    unsigned start = iter->field_index;
-    
-    do {
-        if (iter->pos->tag == tag &&
-            PB_LTYPE(iter->pos->type) != PB_LTYPE_EXTENSION)
-        {
-            return true;
-        }
-        pb_field_next(iter);
-    } while (iter->field_index != start);
-    
-    return false;
-}
-
 /*************************
  * Decode a single field *
  *************************/
@@ -434,11 +417,11 @@ static bool checkreturn decode_callback_field(pb_istream_t *stream, pb_wire_type
         if (!pb_make_string_substream(stream, &substream))
             return false;
         
-        do
+        while (substream.bytes_left)
         {
             if (!pCallback->funcs.decode(&substream, iter->pos, arg))
                 PB_RETURN_ERROR(stream, "callback failed");
-        } while (substream.bytes_left);
+        }
         
         pb_close_string_substream(stream, &substream);
         return true;
@@ -476,136 +459,105 @@ static bool checkreturn decode_field(pb_istream_t *stream, pb_wire_type_t wire_t
     }
 }
 
-/* Default handler for extension fields. Expects a pb_field_t structure
- * in extension->type->arg. */
-static bool checkreturn default_extension_handler(pb_istream_t *stream,
-    pb_extension_t *extension, uint32_t tag, pb_wire_type_t wire_type)
+/* Set field count to zero (or clear has_ field). */
+static void pb_clear_field_count(const pb_field_iterator_t *iter)
 {
-    const pb_field_t *field = (const pb_field_t*)extension->type->arg;
-    pb_field_iterator_t iter;
-    bool dummy;
+    pb_type_t type;
+    type = iter->pos->type;
     
-    if (field->tag != tag)
-        return true;
-    
-    iter.start = field;
-    iter.pos = field;
-    iter.field_index = 0;
-    iter.required_field_index = 0;
-    iter.dest_struct = extension->dest;
-    iter.pData = extension->dest;
-    iter.pSize = &dummy;
-    
-    return decode_field(stream, wire_type, &iter);
-}
+    if (iter->pos->tag == 0)
+        return; /* Empty message type */
 
-/* Try to decode an unknown field as an extension field. Tries each extension
- * decoder in turn, until one of them handles the field or loop ends. */
-static bool checkreturn decode_extension(pb_istream_t *stream,
-    uint32_t tag, pb_wire_type_t wire_type, pb_field_iterator_t *iter)
-{
-    pb_extension_t *extension = *(pb_extension_t* const *)iter->pData;
-    size_t pos = stream->bytes_left;
-    
-    while (extension && pos == stream->bytes_left)
+    if (PB_ATYPE(type) == PB_ATYPE_STATIC)
     {
-        bool status;
-        if (extension->type->decode)
-            status = extension->type->decode(stream, extension, tag, wire_type);
-        else
-            status = default_extension_handler(stream, extension, tag, wire_type);
-
-        if (!status)
-            return false;
-        
-        extension = extension->next;
+        if (PB_HTYPE(type) == PB_HTYPE_OPTIONAL)
+        {
+            *(bool*)iter->pSize = false;
+        }
+        else if (PB_HTYPE(type) == PB_HTYPE_REPEATED)
+        {
+            *(size_t*)iter->pSize = 0;
+        }
     }
-    
-    return true;
 }
 
-/* Step through the iterator until an extension field is found or until all
- * entries have been checked. There can be only one extension field per
- * message. Returns false if no extension field is found. */
-static bool checkreturn find_extension_field(pb_field_iterator_t *iter)
+/* Initialize message field to default value. Recurses on submessages. */
+static void pb_set_field_to_default(const pb_field_iterator_t *iter)
 {
-    unsigned start = iter->field_index;
-    
-    do {
-        if (PB_LTYPE(iter->pos->type) == PB_LTYPE_EXTENSION)
-            return true;
-        pb_field_next(iter);
-    } while (iter->field_index != start);
-    
-    return false;
-}
+    pb_type_t type;
+    type = iter->pos->type;
 
-/* Initialize message fields to default values, recursively */
-static void pb_message_set_to_defaults(const pb_field_t fields[], void *dest_struct)
-{
-    pb_field_iterator_t iter;
-    pb_field_init(&iter, fields, dest_struct);
-    
-    /* Initialize size/has fields and apply default values */
-    do
+    if (iter->pos->tag == 0)
+        return; /* Empty message type */
+
+    /* We only need to initialize static fields.
+     * Furthermore, arrays do not need to be initialized as their length
+     * will be zero by default.
+     */
+    if (PB_ATYPE(type) == PB_ATYPE_STATIC &&
+        PB_HTYPE(type) != PB_HTYPE_REPEATED)
     {
-        pb_type_t type;
-        type = iter.pos->type;
-    
-        if (iter.pos->tag == 0)
-            continue;
-        
-        if (PB_ATYPE(type) == PB_ATYPE_STATIC)
+        if (PB_LTYPE(iter->pos->type) == PB_LTYPE_SUBMESSAGE)
         {
-            /* Initialize the size field for optional/repeated fields to 0. */
-            if (PB_HTYPE(type) == PB_HTYPE_OPTIONAL)
-            {
-                *(bool*)iter.pSize = false;
-            }
-            else if (PB_HTYPE(type) == PB_HTYPE_REPEATED)
-            {
-                *(size_t*)iter.pSize = 0;
-                continue; /* Array is empty, no need to initialize contents */
-            }
-            
-            /* Initialize field contents to default value */
-            if (PB_LTYPE(iter.pos->type) == PB_LTYPE_SUBMESSAGE)
-            {
-                pb_message_set_to_defaults((const pb_field_t *) iter.pos->ptr, iter.pData);
-            }
-            else if (iter.pos->ptr != NULL)
-            {
-                memcpy(iter.pData, iter.pos->ptr, iter.pos->data_size);
-            }
-            else
-            {
-                memset(iter.pData, 0, iter.pos->data_size);
-            }
+            /* Submessage: initialize the fields recursively */
+            pb_field_iterator_t subiter;
+            pb_field_init(&subiter, (const pb_field_t *)iter->pos->ptr, iter->pData);
+            do {
+                pb_clear_field_count(&subiter);
+                pb_set_field_to_default(&subiter);
+            } while (pb_field_next(&subiter));
         }
-        else if (PB_ATYPE(type) == PB_ATYPE_CALLBACK)
+        else if (iter->pos->ptr != NULL)
         {
-            continue; /* Don't overwrite callback */
+            /* Normal field: copy the default value */
+            memcpy(iter->pData, iter->pos->ptr, iter->pos->data_size);
         }
-    } while (pb_field_next(&iter));
+        else
+        {
+            /* Normal field without default value: initialize to zero */
+            memset(iter->pData, 0, iter->pos->data_size);
+        }
+    }
 }
 
 /*********************
  * Decode all fields *
  *********************/
 
-bool checkreturn pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct)
+/* Helper function to initialize fields while advancing iterator */
+static void advance_iterator(pb_field_iterator_t *iter, bool *initialize, bool *current_seen)
+{
+    /* Initialize the fields we didn't decode. */
+    if (*initialize && !*current_seen)
+        pb_set_field_to_default(iter);
+    
+    /* Stop initializing after the first pass through the array */
+    if (!pb_field_next(iter))
+        *initialize = false;
+    
+    /* Clear the field count before decoding */
+    if (*initialize)
+        pb_clear_field_count(iter);
+    
+    /* Reset the flag to indicate that the new field has not been written to yet. */
+    *current_seen = false;
+}
+
+static bool checkreturn pb_decode_inner(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct, bool initialize)
 {
     uint8_t fields_seen[(PB_MAX_REQUIRED_FIELDS + 7) / 8] = {0}; /* Used to check for required fields */
-    uint32_t extension_range_start = 0;
+    bool current_seen = false;
     pb_field_iterator_t iter;
-    
     pb_field_init(&iter, fields, dest_struct);
+    pb_clear_field_count(&iter);
     
     while (stream->bytes_left)
     {
         uint32_t tag;
         pb_wire_type_t wire_type;
         bool eof;
+        unsigned start;
+        bool skip = false;
         
         if (!pb_decode_tag(stream, &wire_type, &tag, &eof))
         {
@@ -615,43 +567,45 @@ bool checkreturn pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[
                 return false;
         }
         
-        if (!pb_field_find(&iter, tag))
+        /* Go through the fields until we either find a match or
+         * wrap around to start. On the first pass, also initialize
+         * any missing fields.
+         * 
+         * The logic here is to avoid unnecessary initialization
+         * in the common case, where all fields occur in the proper
+         * order.
+         */
+        start = iter.field_index;
+        while (iter.pos->tag != tag)
         {
-            /* No match found, check if it matches an extension. */
-            if (tag >= extension_range_start)
+            advance_iterator(&iter, &initialize, &current_seen);
+            
+            if (iter.field_index == start)
             {
-                if (!find_extension_field(&iter))
-                    extension_range_start = (uint32_t)-1;
-                else
-                    extension_range_start = iter.pos->tag;
-                
-                if (tag >= extension_range_start)
-                {
-                    size_t pos = stream->bytes_left;
-                
-                    if (!decode_extension(stream, tag, wire_type, &iter))
-                        return false;
-                    
-                    if (pos != stream->bytes_left)
-                    {
-                        /* The field was handled */
-                        continue;                    
-                    }
-                }
+                skip = true;
+                break;
             }
+        }
         
-            /* No match found, skip data */
+        /* Skip data if field was not found */
+        if (skip)
+        {
             if (!pb_skip_field(stream, wire_type))
                 return false;
+
             continue;
         }
         
+        current_seen = true;
+        
+        /* Keep track if all required fields are present */
         if (PB_HTYPE(iter.pos->type) == PB_HTYPE_REQUIRED
             && iter.required_field_index < PB_MAX_REQUIRED_FIELDS)
         {
             fields_seen[iter.required_field_index >> 3] |= (uint8_t)(1 << (iter.required_field_index & 7));
         }
             
+        /* Finally, decode the field data */
         if (!decode_field(stream, wire_type, &iter))
             return false;
     }
@@ -661,6 +615,9 @@ bool checkreturn pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[
         /* First figure out the number of required fields by
          * seeking to the end of the field array. Usually we
          * are already close to end after decoding.
+         *
+         * Note: this simultaneously initializes any fields
+         * that haven't been already initialized.
          */
         unsigned req_field_count;
         pb_type_t last_type;
@@ -668,7 +625,8 @@ bool checkreturn pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[
         do {
             req_field_count = iter.required_field_index;
             last_type = iter.pos->type;
-        } while (pb_field_next(&iter));
+            advance_iterator(&iter, &initialize, &current_seen);
+        } while (iter.field_index != 0);
         
         /* Fixup if last field was also required. */
         if (PB_HTYPE(last_type) == PB_HTYPE_REQUIRED && iter.pos->tag)
@@ -689,23 +647,14 @@ bool checkreturn pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[
     return true;
 }
 
-bool checkreturn pb_decode(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct)
+bool checkreturn pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct)
 {
-    pb_message_set_to_defaults(fields, dest_struct);
-    return pb_decode_noinit(stream, fields, dest_struct);
+    return pb_decode_inner(stream, fields, dest_struct, false);
 }
 
-bool pb_decode_delimited(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct)
+bool checkreturn pb_decode(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct)
 {
-    pb_istream_t substream;
-    bool status;
-    
-    if (!pb_make_string_substream(stream, &substream))
-        return false;
-    
-    status = pb_decode(&substream, fields, dest_struct);
-    pb_close_string_substream(stream, &substream);
-    return status;
+    return pb_decode_inner(stream, fields, dest_struct, true);
 }
 
 /* Field decoders */
@@ -769,8 +718,7 @@ bool pb_decode_fixed64(pb_istream_t *stream, void *dest)
 bool checkreturn pb_dec_varint(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
     uint64_t value;
-    if (!pb_decode_varint(stream, &value))
-        return false;
+    bool status = pb_decode_varint(stream, &value);
     
     switch (field->data_size)
     {
@@ -781,14 +729,13 @@ bool checkreturn pb_dec_varint(pb_istream_t *stream, const pb_field_t *field, vo
         default: PB_RETURN_ERROR(stream, "invalid data_size");
     }
     
-    return true;
+    return status;
 }
 
 bool checkreturn pb_dec_svarint(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
     int64_t value;
-    if (!pb_decode_svarint(stream, &value))
-        return false;
+    bool status = pb_decode_svarint(stream, &value);
     
     switch (field->data_size)
     {
@@ -797,7 +744,7 @@ bool checkreturn pb_dec_svarint(pb_istream_t *stream, const pb_field_t *field, v
         default: PB_RETURN_ERROR(stream, "invalid data_size");
     }
     
-    return true;
+    return status;
 }
 
 bool checkreturn pb_dec_fixed32(pb_istream_t *stream, const pb_field_t *field, void *dest)
@@ -856,12 +803,7 @@ bool checkreturn pb_dec_submessage(pb_istream_t *stream, const pb_field_t *field
     if (field->ptr == NULL)
         PB_RETURN_ERROR(stream, "invalid field descriptor");
     
-    /* New array entries need to be initialized, while required and optional
-     * submessages have already been initialized in the top-level pb_decode. */
-    if (PB_HTYPE(field->type) == PB_HTYPE_REPEATED)
-        status = pb_decode(&substream, submsg_fields, dest);
-    else
-        status = pb_decode_noinit(&substream, submsg_fields, dest);
+    status = pb_decode(&substream, submsg_fields, dest);
     
     pb_close_string_substream(stream, &substream);
     return status;

pb_decode.h

@@ -1,29 +1,34 @@
-/* pb_decode.h: Functions to decode protocol buffers. Depends on pb_decode.c.
- * The main function is pb_decode. You also need an input stream, and the
- * field descriptions created by nanopb_generator.py.
- */
-
 #ifndef _PB_DECODE_H_
 #define _PB_DECODE_H_
 
+/* pb_decode.h: Functions to decode protocol buffers. Depends on pb_decode.c.
+ * The main function is pb_decode. You will also need to create an input
+ * stream, which is easiest to do with pb_istream_from_buffer().
+ * 
+ * You also need structures and their corresponding pb_field_t descriptions.
+ * These are usually generated from .proto-files with a script.
+ */
+
+#include <stdbool.h>
 #include "pb.h"
 
 #ifdef __cplusplus
 extern "C" {
 #endif
 
-/* Structure for defining custom input streams. You will need to provide
- * a callback function to read the bytes from your storage, which can be
- * for example a file or a network socket.
+/* Lightweight input stream.
+ * You can provide a callback function for reading or use
+ * pb_istream_from_buffer.
  * 
- * The callback must conform to these rules:
- *
+ * Rules for callback:
  * 1) Return false on IO errors. This will cause decoding to abort.
+ * 
  * 2) You can use state to store your own data (e.g. buffer pointer),
- *    and rely on pb_read to verify that no-body reads past bytes_left.
+ * and rely on pb_read to verify that no-body reads past bytes_left.
+ * 
  * 3) Your callback may be used with substreams, in which case bytes_left
- *    is different than from the main stream. Don't use bytes_left to compute
- *    any pointers.
+ * is different than from the main stream. Don't use bytes_left to compute
+ * any pointers.
  */
 struct _pb_istream_t
 {
@@ -45,25 +50,12 @@ struct _pb_istream_t
 #endif
 };
 
-/***************************
- * Main decoding functions *
- ***************************/
- 
-/* Decode a single protocol buffers message from input stream into a C structure.
+pb_istream_t pb_istream_from_buffer(uint8_t *buf, size_t bufsize);
+bool pb_read(pb_istream_t *stream, uint8_t *buf, size_t count);
+
+/* Decode from stream to destination struct.
  * Returns true on success, false on any failure.
  * The actual struct pointed to by dest must match the description in fields.
- * Callback fields of the destination structure must be initialized by caller.
- * All other fields will be initialized by this function.
- *
- * Example usage:
- *    MyMessage msg = {};
- *    uint8_t buffer[64];
- *    pb_istream_t stream;
- *    
- *    // ... read some data into buffer ...
- *
- *    stream = pb_istream_from_buffer(buffer, count);
- *    pb_decode(&stream, MyMessage_fields, &msg);
  */
 bool pb_decode(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct);
 
@@ -71,38 +63,14 @@ bool pb_decode(pb_istream_t *stream, const pb_field_t fields[], void *dest_struc
  * to default values. This is slightly faster if you need no default values
  * and just do memset(struct, 0, sizeof(struct)) yourself.
  *
- * This can also be used for 'merging' two messages, i.e. update only the
- * fields that exist in the new message.
+ * It can also be used to merge fields from a new message into a previously
+ * initialized structure.
  */
 bool pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct);
 
-/* Same as pb_decode, except expects the stream to start with the message size
- * encoded as varint. Corresponds to parseDelimitedFrom() in Google's
- * protobuf API.
+/* --- Helper functions ---
+ * You may want to use these from your caller or callbacks.
  */
-bool pb_decode_delimited(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct);
-
-
-/**************************************
- * Functions for manipulating streams *
- **************************************/
-
-/* Create an input stream for reading from a memory buffer.
- *
- * Alternatively, you can use a custom stream that reads directly from e.g.
- * a file or a network socket.
- */
-pb_istream_t pb_istream_from_buffer(uint8_t *buf, size_t bufsize);
-
-/* Function to read from a pb_istream_t. You can use this if you need to
- * read some custom header data, or to read data in field callbacks.
- */
-bool pb_read(pb_istream_t *stream, uint8_t *buf, size_t count);
-
-
-/************************************************
- * Helper functions for writing field callbacks *
- ************************************************/
 
 /* Decode the tag for the next field in the stream. Gives the wire type and
  * field tag. At end of the message, returns false and sets eof to true. */
@@ -131,10 +99,10 @@ bool pb_decode_fixed64(pb_istream_t *stream, void *dest);
 bool pb_make_string_substream(pb_istream_t *stream, pb_istream_t *substream);
 void pb_close_string_substream(pb_istream_t *stream, pb_istream_t *substream);
 
-
-/*******************************
- * Internal / legacy functions *
- *******************************/
+/* --- Internal functions ---
+ * These functions are not terribly useful for the average library user, but
+ * are exported to make the unit testing and extending nanopb easier.
+ */
 
 #ifdef NANOPB_INTERNALS
 bool pb_dec_varint(pb_istream_t *stream, const pb_field_t *field, void *dest);

pb_encode.c

@@ -28,8 +28,7 @@ static const pb_encoder_t PB_ENCODERS[PB_LTYPES_COUNT] = {
     
     &pb_enc_bytes,
     &pb_enc_string,
-    &pb_enc_submessage,
-    NULL /* extensions */
+    &pb_enc_submessage
 };
 
 /* pb_ostream_t implementation */
@@ -84,7 +83,10 @@ bool checkreturn pb_write(pb_ostream_t *stream, const uint8_t *buf, size_t count
 
 /* Main encoding stuff */
 
-/* Encode a static array. Handles the size calculations and possible packing. */
+/* Callbacks don't need this function because they usually know the data type
+ * without examining the field structure.
+ * Therefore it is static for now.
+ */
 static bool checkreturn encode_array(pb_ostream_t *stream, const pb_field_t *field,
                          const void *pData, size_t count, pb_encoder_t func)
 {
@@ -95,7 +97,6 @@ static bool checkreturn encode_array(pb_ostream_t *stream, const pb_field_t *fie
     if (count == 0)
         return true;
     
-    /* We always pack arrays if the datatype allows it. */
     if (PB_LTYPE(field->type) <= PB_LTYPE_LAST_PACKABLE)
     {
         if (!pb_encode_tag(stream, PB_WT_STRING, field->tag))
@@ -111,7 +112,7 @@ static bool checkreturn encode_array(pb_ostream_t *stream, const pb_field_t *fie
             size = 8 * count;
         }
         else
-        { 
+        {
             pb_ostream_t sizestream = PB_OSTREAM_SIZING;
             p = pData;
             for (i = 0; i < count; i++)
@@ -154,21 +155,13 @@ static bool checkreturn encode_array(pb_ostream_t *stream, const pb_field_t *fie
     return true;
 }
 
-/* Encode a field with static allocation, i.e. one whose data is stored
- * in the structure itself. */
-static bool checkreturn encode_static_field(pb_ostream_t *stream,
-    const pb_field_t *field, const void *pData)
+bool checkreturn encode_static_field(pb_ostream_t *stream, const pb_field_t *field, const void *pData)
 {
     pb_encoder_t func;
     const void *pSize;
-    bool dummy = true;
     
     func = PB_ENCODERS[PB_LTYPE(field->type)];
-    
-    if (field->size_offset)
-        pSize = (const char*)pData + field->size_offset;
-    else
-        pSize = &dummy;
+    pSize = (const char*)pData + field->size_offset;
     
     switch (PB_HTYPE(field->type))
     {
@@ -202,10 +195,7 @@ static bool checkreturn encode_static_field(pb_ostream_t *stream,
     return true;
 }
 
-/* Encode a field with callback semantics. This means that a user function is
- * called to provide and encode the actual data. */
-static bool checkreturn encode_callback_field(pb_ostream_t *stream,
-    const pb_field_t *field, const void *pData)
+bool checkreturn encode_callback_field(pb_ostream_t *stream, const pb_field_t *field, const void *pData)
 {
     const pb_callback_t *callback = (const pb_callback_t*)pData;
     
@@ -223,57 +213,6 @@ static bool checkreturn encode_callback_field(pb_ostream_t *stream,
     return true;
 }
 
-/* Encode a single field of any callback or static type. */
-static bool checkreturn encode_field(pb_ostream_t *stream,
-    const pb_field_t *field, const void *pData)
-{
-    switch (PB_ATYPE(field->type))
-    {
-        case PB_ATYPE_STATIC:
-            return encode_static_field(stream, field, pData);
-        
-        case PB_ATYPE_CALLBACK:
-            return encode_callback_field(stream, field, pData);
-        
-        default:
-            PB_RETURN_ERROR(stream, "invalid field type");
-    }
-}
-
-/* Default handler for extension fields. Expects to have a pb_field_t
- * pointer in the extension->type->arg field. */
-static bool checkreturn default_extension_handler(pb_ostream_t *stream,
-    const pb_extension_t *extension)
-{
-    const pb_field_t *field = (const pb_field_t*)extension->type->arg;
-    return encode_field(stream, field, extension->dest);
-}
-
-/* Walk through all the registered extensions and give them a chance
- * to encode themselves. */
-static bool checkreturn encode_extension_field(pb_ostream_t *stream,
-    const pb_field_t *field, const void *pData)
-{
-    const pb_extension_t *extension = *(const pb_extension_t* const *)pData;
-    UNUSED(field);
-    
-    while (extension)
-    {
-        bool status;
-        if (extension->type->encode)
-            status = extension->type->encode(stream, extension);
-        else
-            status = default_extension_handler(stream, extension);
-
-        if (!status)
-            return false;
-        
-        extension = extension->next;
-    }
-    
-    return true;
-}
-
 bool checkreturn pb_encode(pb_ostream_t *stream, const pb_field_t fields[], const void *src_struct)
 {
     const pb_field_t *field = fields;
@@ -291,18 +230,21 @@ bool checkreturn pb_encode(pb_ostream_t *stream, const pb_field_t fields[], cons
         {
             prev_size *= field->array_size;
         }
-        
-        if (PB_LTYPE(field->type) == PB_LTYPE_EXTENSION)
+                
+        switch (PB_ATYPE(field->type))
         {
-            /* Special case for the extension field placeholder */
-            if (!encode_extension_field(stream, field, pData))
-                return false;
-        }
-        else
-        {
-            /* Regular field */
-            if (!encode_field(stream, field, pData))
-                return false;
+            case PB_ATYPE_STATIC:
+                if (!encode_static_field(stream, field, pData))
+                    return false;
+                break;
+            
+            case PB_ATYPE_CALLBACK:
+                if (!encode_callback_field(stream, field, pData))
+                    return false;
+                break;
+            
+            default:
+                PB_RETURN_ERROR(stream, "invalid field type");
         }
     
         field++;
@@ -311,11 +253,6 @@ bool checkreturn pb_encode(pb_ostream_t *stream, const pb_field_t fields[], cons
     return true;
 }
 
-bool pb_encode_delimited(pb_ostream_t *stream, const pb_field_t fields[], const void *src_struct)
-{
-    return pb_encode_submessage(stream, fields, src_struct);
-}
-
 /* Helper functions */
 bool checkreturn pb_encode_varint(pb_ostream_t *stream, uint64_t value)
 {
@@ -524,16 +461,8 @@ bool checkreturn pb_enc_bytes(pb_ostream_t *stream, const pb_field_t *field, con
 
 bool checkreturn pb_enc_string(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
-    /* strnlen() is not always available, so just use a for-loop */
-    size_t size = 0;
-    const char *p = (const char*)src;
-    while (size < field->data_size && *p != '\0')
-    {
-        size++;
-        p++;
-    }
-
-    return pb_encode_string(stream, (const uint8_t*)src, size);
+    UNUSED(field);
+    return pb_encode_string(stream, (const uint8_t*)src, strlen((const char*)src));
 }
 
 bool checkreturn pb_enc_submessage(pb_ostream_t *stream, const pb_field_t *field, const void *src)

pb_encode.h

@@ -1,28 +1,34 @@
-/* pb_encode.h: Functions to encode protocol buffers. Depends on pb_encode.c.
- * The main function is pb_encode. You also need an output stream, and the
- * field descriptions created by nanopb_generator.py.
- */
-
 #ifndef _PB_ENCODE_H_
 #define _PB_ENCODE_H_
 
+/* pb_encode.h: Functions to encode protocol buffers. Depends on pb_encode.c.
+ * The main function is pb_encode. You also need an output stream, structures
+ * and their field descriptions (just like with pb_decode).
+ */
+
+#include <stdbool.h>
 #include "pb.h"
 
 #ifdef __cplusplus
 extern "C" {
 #endif
 
-/* Structure for defining custom output streams. You will need to provide
- * a callback function to write the bytes to your storage, which can be
- * for example a file or a network socket.
- *
- * The callback must conform to these rules:
+/* Lightweight output stream.
+ * You can provide callback for writing or use pb_ostream_from_buffer.
+ * 
+ * Alternatively, callback can be NULL in which case the stream will just
+ * count the number of bytes that would have been written. In this case
+ * max_size is not checked.
  *
+ * Rules for callback:
  * 1) Return false on IO errors. This will cause encoding to abort.
+ * 
  * 2) You can use state to store your own data (e.g. buffer pointer).
+ * 
  * 3) pb_write will update bytes_written after your callback runs.
- * 4) Substreams will modify max_size and bytes_written. Don't use them
- *    to calculate any pointers.
+ * 
+ * 4) Substreams will modify max_size and bytes_written. Don't use them to
+ * calculate any pointers.
  */
 struct _pb_ostream_t
 {
@@ -37,84 +43,42 @@ struct _pb_ostream_t
 #else
     bool (*callback)(pb_ostream_t *stream, const uint8_t *buf, size_t count);
 #endif
-    void *state;          /* Free field for use by callback implementation. */
-    size_t max_size;      /* Limit number of output bytes written (or use SIZE_MAX). */
-    size_t bytes_written; /* Number of bytes written so far. */
+    void *state; /* Free field for use by callback implementation */
+    size_t max_size; /* Limit number of output bytes written (or use SIZE_MAX). */
+    size_t bytes_written;
     
 #ifndef PB_NO_ERRMSG
     const char *errmsg;
 #endif
 };
 
-/***************************
- * Main encoding functions *
- ***************************/
-
-/* Encode a single protocol buffers message from C structure into a stream.
- * Returns true on success, false on any failure.
- * The actual struct pointed to by src_struct must match the description in fields.
- * All required fields in the struct are assumed to have been filled in.
- *
- * Example usage:
- *    MyMessage msg = {};
- *    uint8_t buffer[64];
- *    pb_ostream_t stream;
- *
- *    msg.field1 = 42;
- *    stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
- *    pb_encode(&stream, MyMessage_fields, &msg);
- */
-bool pb_encode(pb_ostream_t *stream, const pb_field_t fields[], const void *src_struct);
-
-/* Same as pb_encode, but prepends the length of the message as a varint.
- * Corresponds to writeDelimitedTo() in Google's protobuf API.
- */
-bool pb_encode_delimited(pb_ostream_t *stream, const pb_field_t fields[], const void *src_struct);
-
-/**************************************
- * Functions for manipulating streams *
- **************************************/
-
-/* Create an output stream for writing into a memory buffer.
- * The number of bytes written can be found in stream.bytes_written after
- * encoding the message.
- *
- * Alternatively, you can use a custom stream that writes directly to e.g.
- * a file or a network socket.
- */
 pb_ostream_t pb_ostream_from_buffer(uint8_t *buf, size_t bufsize);
+bool pb_write(pb_ostream_t *stream, const uint8_t *buf, size_t count);
 
-/* Pseudo-stream for measuring the size of a message without actually storing
- * the encoded data.
- * 
- * Example usage:
- *    MyMessage msg = {};
- *    pb_ostream_t stream = PB_OSTREAM_SIZING;
- *    pb_encode(&stream, MyMessage_fields, &msg);
- *    printf("Message size is %d\n", stream.bytes_written);
- */
+/* Stream type for use in computing message sizes */
 #ifndef PB_NO_ERRMSG
 #define PB_OSTREAM_SIZING {0,0,0,0,0}
 #else
 #define PB_OSTREAM_SIZING {0,0,0,0}
 #endif
 
-/* Function to write into a pb_ostream_t stream. You can use this if you need
- * to append or prepend some custom headers to the message.
+/* Encode struct to given output stream.
+ * Returns true on success, false on any failure.
+ * The actual struct pointed to by src_struct must match the description in fields.
+ * All required fields in the struct are assumed to have been filled in.
  */
-bool pb_write(pb_ostream_t *stream, const uint8_t *buf, size_t count);
+bool pb_encode(pb_ostream_t *stream, const pb_field_t fields[], const void *src_struct);
 
+/* --- Helper functions ---
+ * You may want to use these from your caller or callbacks.
+ */
 
-/************************************************
- * Helper functions for writing field callbacks *
- ************************************************/
-
-/* Encode field header based on type and field number defined in the field
- * structure. Call this from the callback before writing out field contents. */
+/* Encode field header based on LTYPE and field number defined in the field structure.
+ * Call this from the callback before writing out field contents. */
 bool pb_encode_tag_for_field(pb_ostream_t *stream, const pb_field_t *field);
 
-/* Encode field header by manually specifing wire type. You need to use this
- * if you want to write out packed arrays from a callback field. */
+/* Encode field header by manually specifing wire type. You need to use this if
+ * you want to write out packed arrays from a callback field. */
 bool pb_encode_tag(pb_ostream_t *stream, pb_wire_type_t wiretype, uint32_t field_number);
 
 /* Encode an integer in the varint format.
@@ -137,16 +101,15 @@ bool pb_encode_fixed32(pb_ostream_t *stream, const void *value);
 bool pb_encode_fixed64(pb_ostream_t *stream, const void *value);
 
 /* Encode a submessage field.
- * You need to pass the pb_field_t array and pointer to struct, just like
- * with pb_encode(). This internally encodes the submessage twice, first to
- * calculate message size and then to actually write it out.
+ * You need to pass the pb_field_t array and pointer to struct, just like with pb_encode().
+ * This internally encodes the submessage twice, first to calculate message size and then to actually write it out.
  */
 bool pb_encode_submessage(pb_ostream_t *stream, const pb_field_t fields[], const void *src_struct);
 
-
-/*******************************
- * Internal / legacy functions *
- *******************************/
+/* --- Internal functions ---
+ * These functions are not terribly useful for the average library user, but
+ * are exported to make the unit testing and extending nanopb easier.
+ */
 
 #ifdef NANOPB_INTERNALS
 bool pb_enc_varint(pb_ostream_t *stream, const pb_field_t *field, const void *src);

tests/Makefile

@@ -8,7 +8,7 @@ TESTS=  decode_unittests encode_unittests \
 	test_decode_callbacks test_encode_callbacks \
 	test_missing_fields test_no_messages test_funny_name \
 	test_multiple_files test_cxxcompile test_options \
-	bc_encode bc_decode test_encode_extensions test_decode_extensions
+	bc_encode bc_decode
 
 # More strict checks for the core part of nanopb
 CC_VERSION=$(shell $(CC) -v 2>&1)
@@ -84,8 +84,6 @@ test_no_messages: no_messages.pb.h no_messages.pb.c no_messages.pb.o
 test_funny_name: funny-proto+name.pb.h funny-proto+name.pb.o
 bc_encode: bc_alltypes.pb.o pb_encode.o bc_encode.o
 bc_decode: bc_alltypes.pb.o pb_decode.o bc_decode.o
-test_encode_extensions: test_encode_extensions.c pb_encode.o alltypes.pb.o extensions.pb.o
-test_decode_extensions: test_decode_extensions.c pb_decode.o alltypes.pb.o extensions.pb.o
 
 %.pb: %.proto
 	protoc -I. -I../generator -I/usr/include -o$@ $<
@@ -125,10 +123,8 @@ run_unittests: $(TESTS)
 	./test_encode3 1 | ./test_decode3 1
 	./test_encode3 1 | protoc --decode=AllTypes -I. -I../generator -I/usr/include alltypes.proto >/dev/null
 	./test_encode3_buf 1 | ./test_decode3_buf 1
-	./test_decode3 < alltypes_with_extra_fields.pb
 	./bc_encode | ./bc_decode
-	./test_encode_extensions | ./test_decode_extensions
-
+	
 	./test_missing_fields
 
 test_options: options.pb.h options.expected options.pb.o

tests/alltypes.proto

@@ -86,8 +86,5 @@ message AllTypes {
     // Just to make sure that the size of the fields has been calculated
     // properly, i.e. otherwise a bug in last field might not be detected.
     required int32      end = 99;
-
-
-    extensions 200 to 255;
 }
 

tests/callbacks.proto

@@ -11,6 +11,5 @@ message TestMessage {
     repeated fixed32 fixed32value = 3;
     repeated fixed64 fixed64value = 4;
     optional SubMessage submsg = 5;
-    repeated string repeatedstring = 6;
 }
 

tests/decode_unittests.c

@@ -289,16 +289,6 @@ int main()
         TEST((s = S("\x08"), !pb_decode(&s, IntegerArray_fields, &dest)))
     }
     
-    {
-        pb_istream_t s;
-        IntegerContainer dest = {};
-        
-        COMMENT("Testing pb_decode_delimited")
-        TEST((s = S("\x09\x0A\x07\x0A\x05\x01\x02\x03\x04\x05"),
-              pb_decode_delimited(&s, IntegerContainer_fields, &dest)) &&
-              dest.submsg.data_count == 5)
-    }
-    
     if (status != 0)
         fprintf(stdout, "\n\nSome tests FAILED!\n");
     

tests/encode_unittests.c

@@ -180,14 +180,12 @@ int main()
     {
         uint8_t buffer[30];
         pb_ostream_t s;
-        char value[30] = "xyzzy";
+        char value[] = "xyzzy";
         
         COMMENT("Test pb_enc_string")
-        TEST(WRITES(pb_enc_string(&s, &StringMessage_fields[0], &value), "\x05xyzzy"))
+        TEST(WRITES(pb_enc_string(&s, NULL, &value), "\x05xyzzy"))
         value[0] = '\0';
-        TEST(WRITES(pb_enc_string(&s, &StringMessage_fields[0], &value), "\x00"))
-        memset(value, 'x', 30);
-        TEST(WRITES(pb_enc_string(&s, &StringMessage_fields[0], &value), "\x0Axxxxxxxxxx"))
+        TEST(WRITES(pb_enc_string(&s, NULL, &value), "\x00"))
     }
     
     {
@@ -244,16 +242,6 @@ int main()
                     "\x0A\x07\x0A\x05\x01\x02\x03\x04\x05"))
     }
     
-    {
-        uint8_t buffer[20];
-        pb_ostream_t s;
-        IntegerContainer msg = {{5, {1,2,3,4,5}}};
-        
-        COMMENT("Test pb_encode_delimited.")
-        TEST(WRITES(pb_encode_delimited(&s, IntegerContainer_fields, &msg),
-                    "\x09\x0A\x07\x0A\x05\x01\x02\x03\x04\x05"))
-    }
-    
     {
         uint8_t buffer[10];
         pb_ostream_t s;

tests/extensions.options

@@ -1 +0,0 @@
-* max_size:16

tests/extensions.proto

@@ -1,17 +0,0 @@
-import 'alltypes.proto';
-
-extend AllTypes {
-    optional int32 AllTypes_extensionfield1 = 255;
-}
-
-message ExtensionMessage {
-    extend AllTypes {
-        optional ExtensionMessage AllTypes_extensionfield2 = 254;
-	required ExtensionMessage AllTypes_extensionfield3 = 253;
-	repeated ExtensionMessage AllTypes_extensionfield4 = 252;
-    }
-    
-    required string test1 = 1;
-    required int32 test2 = 2;
-}
-

tests/test_decode_callbacks.c

@@ -83,8 +83,6 @@ int main()
     testmessage.fixed32value.arg = "fixed32value: %ld\n";
     testmessage.fixed64value.funcs.decode = &print_fixed64;
     testmessage.fixed64value.arg = "fixed64value: %lld\n";
-    testmessage.repeatedstring.funcs.decode = &print_string;
-    testmessage.repeatedstring.arg = "repeatedstring: \"%s\"\n";
     
     if (!pb_decode(&stream, TestMessage_fields, &testmessage))
         return 1;

tests/test_decode_extensions.c

@@ -1,43 +0,0 @@
-/* Test decoding of extension fields. */
-
-#include <stdio.h>
-#include <string.h>
-#include <stdlib.h>
-#include <pb_decode.h>
-#include "alltypes.pb.h"
-#include "extensions.pb.h"
-
-#define TEST(x) if (!(x)) { \
-    printf("Test " #x " failed.\n"); \
-    return 2; \
-    }
-
-int main(int argc, char **argv)
-{
-    uint8_t buffer[1024];
-    size_t count = fread(buffer, 1, sizeof(buffer), stdin);
-    pb_istream_t stream = pb_istream_from_buffer(buffer, count);
-    
-    AllTypes alltypes = {};
-    
-    int32_t extensionfield1;
-    pb_extension_t ext1 = {&AllTypes_extensionfield1, &extensionfield1, NULL};
-    alltypes.extensions = &ext1;
-        
-    ExtensionMessage extensionfield2 = {};
-    pb_extension_t ext2 = {&ExtensionMessage_AllTypes_extensionfield2, &extensionfield2, NULL};
-    ext1.next = &ext2;
-    
-    if (!pb_decode(&stream, AllTypes_fields, &alltypes))
-    {
-        printf("Parsing failed: %s\n", PB_GET_ERROR(&stream));
-        return 1;
-    }
-
-    TEST(extensionfield1 == 12345)
-    TEST(strcmp(extensionfield2.test1, "test") == 0)
-    TEST(extensionfield2.test2 == 54321)
-    
-    return 0;
-}
-

tests/test_encode_callbacks.c

@@ -41,22 +41,6 @@ bool encode_fixed64(pb_ostream_t *stream, const pb_field_t *field, void * const
     return pb_encode_fixed64(stream, &value);
 }
 
-bool encode_repeatedstring(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
-{
-    char *str[4] = {"Hello world!", "", "Test", "Test2"};
-    int i;
-    
-    for (i = 0; i < 4; i++)
-    {
-        if (!pb_encode_tag_for_field(stream, field))
-            return false;
-        
-        if (!pb_encode_string(stream, (uint8_t*)str[i], strlen(str[i])))
-            return false;
-    }
-    return true;
-}
-
 int main()
 {
     uint8_t buffer[1024];
@@ -73,8 +57,6 @@ int main()
     testmessage.submsg.int32value.funcs.encode = &encode_int32;
     testmessage.submsg.fixed32value.funcs.encode = &encode_fixed32;
     testmessage.submsg.fixed64value.funcs.encode = &encode_fixed64;
-
-    testmessage.repeatedstring.funcs.encode = &encode_repeatedstring;
     
     if (!pb_encode(&stream, TestMessage_fields, &testmessage))
         return 1;

tests/test_encode_extensions.c

@@ -1,38 +0,0 @@
-/* Tests extension fields.
- */
-
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-#include <pb_encode.h>
-#include "alltypes.pb.h"
-#include "extensions.pb.h"
-
-int main(int argc, char **argv)
-{
-    AllTypes alltypes = {};
-
-    int32_t extensionfield1 = 12345;
-    pb_extension_t ext1 = {&AllTypes_extensionfield1, &extensionfield1, NULL};
-    alltypes.extensions = &ext1;
-
-    ExtensionMessage extensionfield2 = {"test", 54321};
-    pb_extension_t ext2 = {&ExtensionMessage_AllTypes_extensionfield2, &extensionfield2, NULL};
-    ext1.next = &ext2;
-    
-    uint8_t buffer[1024];
-    pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
-    
-    /* Now encode it and check if we succeeded. */
-    if (pb_encode(&stream, AllTypes_fields, &alltypes))
-    {
-        fwrite(buffer, 1, stream.bytes_written, stdout);
-        return 0; /* Success */
-    }
-    else
-    {
-        fprintf(stderr, "Encoding failed: %s\n", PB_GET_ERROR(&stream));
-        return 1; /* Failure */
-    }
-}
-

tests/unittestproto.proto

@@ -8,10 +8,6 @@ message FloatArray {
     repeated float data = 1 [(nanopb).max_count = 10];
 }
 
-message StringMessage {
-    required string data = 1 [(nanopb).max_size = 10];
-}
-
 message CallbackArray {
     // We cheat a bit and use this message for testing other types, too.
     // Nanopb does not care about the actual defined data type for callback