Publishing nanopb-0.3.0

The version in PB_PROTO_HEADER_VERSION can be bumped whenever there
is a breaking change to the generated files, and it will then alert
to the difference.

Update issue 129
Status: FixedInGit

CHANGELOG.txt

@@ -1,3 +1,42 @@
+nanopb-0.3.0 (2014-08-26)
+ NOTE: See docs/migration.html or online at
+ http://koti.kapsi.fi/~jpa/nanopb/docs/migration.html
+ for changes in this version. Most importantly, you need to add
+ pb_common.c to the list of files to compile.
+
+ Separated field iterator logic to pb_common.c (issue 128)
+ Change the _count fields to use pb_size_t datatype (issue 82)
+ Added PB_ prefix to macro names (issue 106)
+ Added #if version guard to generated files (issue 129)
+ Added migration document
+
+nanopb-0.2.9 (2014-08-09)
+ NOTE: If you are using the -e option with the generator, you have
+ to prepend . to the argument to get the same behaviour as before.
+
+ Do not automatically add a dot with generator -e option. (issue 122)
+ Fix problem with .options file and extension fields. (issue 125)
+ Don't use SIZE_MAX macro, as it is not in C89. (issue 120)
+ Generate #defines for initializing message structures. (issue 79)
+ Add skip_message option to generator. (issue 121)
+ Add PB_PACKED_STRUCT support for Keil MDK-ARM toolchain (issue 119)
+ Give better messages about the .options file path. (issue 124)
+ Improved tests
+
+nanopb-0.2.8 (2014-05-20)
+ Fix security issue with PB_ENABLE_MALLOC. (issue 117)
+ Add option to not add timestamps to .pb.h and .pb.c preambles. (issue 115)
+ Documentation updates
+ Improved tests
+
+nanopb-0.2.7 (2014-04-07)
+ Fix bug with default values for extension fields (issue 111)
+ Fix some MISRA-C warnings (issue 91)
+ Implemented optional malloc() support (issue 80)
+ Changed pointer-type bytes field datatype
+ Add a "found" field to pb_extension_t (issue 112)
+ Add convenience function pb_get_encoded_size() (issue 16)
+
 nanopb-0.2.6 (2014-02-15)
  Fix generator error with bytes callback fields (issue 99)
  Fix warnings about large integer constants (issue 102)

docs/Makefile

@@ -1,4 +1,4 @@
-all: index.html concepts.html reference.html security.html \
+all: index.html concepts.html reference.html security.html migration.html \
 	generator_flow.png
 
 %.png: %.svg

docs/index.rst

@@ -47,7 +47,6 @@ Features and limitations
 
 **Limitations**
 
-#) User must provide callbacks when decoding arrays or strings without maximum size. Malloc support could be added as a separate module.
 #) Some speed has been sacrificed for code size.
 #) Encoding is focused on writing to streams. For memory buffers only it could be made more efficient.
 #) The deprecated Protocol Buffers feature called "groups" is not supported.

docs/menu.rst

@@ -4,8 +4,10 @@
     2) `Concepts`_
     3) `API reference`_
     4) `Security model`_
+    5) `Migration from older versions`_
     
 .. _`Overview`: index.html
 .. _`Concepts`: concepts.html
 .. _`API reference`: reference.html
 .. _`Security model`: security.html
+.. _`Migration from older versions`: migration.html

docs/migration.rst

@@ -0,0 +1,239 @@
+=====================================
+Nanopb: Migration from older versions
+=====================================
+
+.. include :: menu.rst
+
+This document details all the breaking changes that have been made to nanopb
+since its initial release. For each change, the rationale and required
+modifications of user applications are explained. Also any error indications
+are included, in order to make it easier to find this document.
+
+.. contents ::
+
+Nanopb-0.3.0 (2014-08-26)
+=========================
+
+Separate field iterator logic to pb_common.c
+--------------------------------------------
+**Rationale:** Originally, the field iteration logic was simple enough to be
+duplicated in *pb_decode.c* and *pb_encode.c*. New field types have made the
+logic more complex, which required the creation of a new file to contain the
+common functionality.
+
+**Changes:** There is a new file, *pb_common.c*, which must be included in
+builds.
+
+**Required actions:** Add *pb_common.c* to build rules. This file is always
+required. Either *pb_decode.c* or *pb_encode.c* can still be left out if some
+functionality is not needed.
+
+**Error indications:** Linker error: undefined reference to
+*pb_field_iter_begin*, *pb_field_iter_next* or similar.
+
+Change data type of field counts to pb_size_t
+---------------------------------------------
+**Rationale:** Often nanopb is used with small arrays, such as 255 items or
+less. Using a full *size_t* field to store the array count wastes memory if
+there are many arrays. There already exists parameters *PB_FIELD_16BIT* and
+*PB_FIELD_32BIT* which tell nanopb what is the maximum size of arrays in use.
+
+**Changes:** Generator will now use *pb_size_t* for the array *_count* fields.
+The size of the type will be controlled by the *PB_FIELD_16BIT* and
+*PB_FIELD_32BIT* compilation time options.
+
+**Required actions:** Regenerate all *.pb.h* files. In some cases casts to the
+*pb_size_t* type may need to be added in the user code when accessing the
+*_count* fields.
+
+**Error indications:** Incorrect data at runtime, crashes. But note that other
+changes in the same version already require regenerating the files and have
+better indications of errors, so this is only an issue for development
+versions.
+
+Renamed some macros and identifiers
+-----------------------------------
+**Rationale:** Some names in nanopb core were badly chosen and conflicted with
+ISO C99 reserved names or lacked a prefix. While they haven't caused trouble
+so far, it is reasonable to switch to non-conflicting names as these are rarely
+used from user code.
+
+**Changes:** The following identifier names have changed:
+
+  * Macros:
+  
+    * STATIC_ASSERT(x) -> PB_STATIC_ASSERT(x)
+    * UNUSED(x) -> PB_UNUSED(x)
+  
+  * Include guards:
+  
+    * _PB_filename_ -> PB_filename_INCLUDED
+  
+  * Structure forward declaration tags:
+  
+    * _pb_field_t -> pb_field_s
+    * _pb_bytes_array_t -> pb_bytes_array_s
+    * _pb_callback_t -> pb_callback_s
+    * _pb_extension_type_t -> pb_extension_type_s
+    * _pb_extension_t -> pb_extension_s
+    * _pb_istream_t -> pb_istream_s
+    * _pb_ostream_t -> pb_ostream_s
+
+**Required actions:** Regenerate all *.pb.c* files. If you use any of the above
+identifiers in your application code, perform search-replace to the new name.
+
+**Error indications:** Compiler errors on lines with the macro/type names.
+
+Nanopb-0.2.9 (2014-08-09)
+=========================
+
+Change semantics of generator -e option
+---------------------------------------
+**Rationale:** Some compilers do not accept filenames with two dots (like
+in default extension .pb.c). The *-e* option to the generator allowed changing
+the extension, but not skipping the extra dot.
+
+**Changes:** The *-e* option in generator will no longer add the prepending
+dot. The default value has been adjusted accordingly to *.pb.c* to keep the
+default behaviour the same as before.
+
+**Required actions:** Only if using the generator -e option. Add dot before
+the parameter value on the command line.
+
+**Error indications:** File not found when trying to compile generated files.
+
+Nanopb-0.2.7 (2014-04-07)
+=========================
+
+Changed pointer-type bytes field datatype
+-----------------------------------------
+**Rationale:** In the initial pointer encoding support since nanopb-0.2.5,
+the bytes type used a separate *pb_bytes_ptr_t* type to represent *bytes*
+fields. This made it easy to encode data from a separate, user-allocated
+buffer. However, it made the internal logic more complex and was inconsistent
+with the other types.
+
+**Changes:** Dynamically allocated bytes fields now have the *pb_bytes_array_t*
+type, just like statically allocated ones.
+
+**Required actions:** Only if using pointer-type fields with the bytes datatype.
+Change any access to *msg->field.size* to *msg->field->size*. Change any
+allocation to reserve space of amount *PB_BYTES_ARRAY_T_ALLOCSIZE(n)*. If the
+data pointer was begin assigned from external source, implement the field using
+a callback function instead.
+
+**Error indications:** Compiler error: unknown type name *pb_bytes_ptr_t*.
+
+Nanopb-0.2.4 (2013-11-07)
+=========================
+
+Remove the NANOPB_INTERNALS compilation option
+----------------------------------------------
+**Rationale:** Having the option in the headers required the functions to
+be non-static, even if the option is not used. This caused errors on some
+static analysis tools.
+
+**Changes:** The *#ifdef* and associated functions were removed from the
+header.
+
+**Required actions:** Only if the *NANOPB_INTERNALS* option was previously
+used. Actions are as listed under nanopb-0.1.3 and nanopb-0.1.6.
+
+**Error indications:** Compiler warning: implicit declaration of function
+*pb_dec_string*, *pb_enc_string*, or similar.
+
+Nanopb-0.2.1 (2013-04-14)
+=========================
+
+Callback function signature
+---------------------------
+**Rationale:** Previously the auxilary data to field callbacks was passed
+as *void\**. This allowed passing of any data, but made it unnecessarily
+complex to return a pointer from callback.
+
+**Changes:** The callback function parameter was changed to *void\**.
+
+**Required actions:** You can continue using the old callback style by
+defining *PB_OLD_CALLBACK_STYLE*. Recommended action is to:
+
+  * Change the callback signatures to contain *void\** for decoders and
+    *void \* const \** for encoders.
+  * Change the callback function body to use *\*arg* instead of *arg*.
+
+**Error indications:** Compiler warning: assignment from incompatible
+pointer type, when initializing *funcs.encode* or *funcs.decode*.
+
+Nanopb-0.2.0 (2013-03-02)
+=========================
+
+Reformatted generated .pb.c file using macros
+---------------------------------------------
+**Rationale:** Previously the generator made a list of C *pb_field_t*
+initializers in the .pb.c file. This led to a need to regenerate all .pb.c
+files after even small changes to the *pb_field_t* definition.
+
+**Changes:** Macros were added to pb.h which allow for cleaner definition
+of the .pb.c contents. By changing the macro definitions, changes to the
+field structure are possible without breaking compatibility with old .pb.c
+files.
+
+**Required actions:** Regenerate all .pb.c files from the .proto sources.
+
+**Error indications:** Compiler warning: implicit declaration of function
+*pb_delta_end*.
+
+Changed pb_type_t definitions
+-----------------------------
+**Rationale:** The *pb_type_t* was previously an enumeration type. This
+caused warnings on some compilers when using bitwise operations to set flags
+inside the values.
+
+**Changes:** The *pb_type_t* was changed to *typedef uint8_t*. The values
+were changed to *#define*. Some value names were changed for consistency.
+
+**Required actions:** Only if you directly access the `pb_field_t` contents
+in your own code, something which is not usually done. Needed changes:
+
+  * Change *PB_HTYPE_ARRAY* to *PB_HTYPE_REPEATED*.
+  * Change *PB_HTYPE_CALLBACK* to *PB_ATYPE()* and *PB_ATYPE_CALLBACK*.
+
+**Error indications:** Compiler error: *PB_HTYPE_ARRAY* or *PB_HTYPE_CALLBACK*
+undeclared.
+
+Nanopb-0.1.6 (2012-09-02)
+=========================
+
+Refactored field decoder interface
+----------------------------------
+**Rationale:** Similarly to field encoders in nanopb-0.1.3.
+
+**Changes:** New functions with names *pb_decode_\** were added.
+
+**Required actions:** By defining NANOPB_INTERNALS, you can still keep using
+the old functions. Recommended action is to replace any calls with the newer
+*pb_decode_\** equivalents.
+
+**Error indications:** Compiler warning: implicit declaration of function
+*pb_dec_string*, *pb_dec_varint*, *pb_dec_submessage* or similar.
+
+Nanopb-0.1.3 (2012-06-12)
+=========================
+
+Refactored field encoder interface
+----------------------------------
+**Rationale:** The old *pb_enc_\** functions were designed mostly for the
+internal use by the core. Because they are internally accessed through
+function pointers, their signatures had to be common. This led to a confusing
+interface for external users.
+
+**Changes:** New functions with names *pb_encode_\** were added. These have
+easier to use interfaces. The old functions are now only thin wrappers for
+the new interface.
+
+**Required actions:** By defining NANOPB_INTERNALS, you can still keep using
+the old functions. Recommended action is to replace any calls with the newer
+*pb_encode_\** equivalents.
+
+**Error indications:** Compiler warning: implicit declaration of function
+*pb_enc_string*, *pb_enc_varint, *pb_enc_submessage* or similar.
+

docs/reference.rst

@@ -24,10 +24,6 @@ __BIG_ENDIAN__                 Set this if your platform stores integers and
                                floats in big-endian format. Mixed-endian
                                systems (different layout for ints and floats)
                                are currently not supported.
-NANOPB_INTERNALS               Set this to expose the field encoder functions
-                               that are hidden since nanopb-0.1.3. Starting
-                               with nanopb-0.2.4, this flag does nothing. Use
-                               the newer functions that have better interface.
 PB_ENABLE_MALLOC               Set this to enable dynamic allocation support
                                in the decoder.
 PB_MAX_REQUIRED_FIELDS         Maximum number of required fields to check for

examples/network_server/client.c

@@ -23,9 +23,13 @@
 #include "fileproto.pb.h"
 #include "common.h"
 
+/* This callback function will be called once for each filename received
+ * from the server. The filenames will be printed out immediately, so that
+ * no memory has to be allocated for them.
+ */
 bool printfile_callback(pb_istream_t *stream, const pb_field_t *field, void **arg)
 {
-    FileInfo fileinfo;
+    FileInfo fileinfo = {};
     
     if (!pb_decode(stream, FileInfo_fields, &fileinfo))
         return false;
@@ -35,14 +39,20 @@ bool printfile_callback(pb_istream_t *stream, const pb_field_t *field, void **ar
     return true;
 }
 
+/* This function sends a request to socket 'fd' to list the files in
+ * directory given in 'path'. The results received from server will
+ * be printed to stdout.
+ */
 bool listdir(int fd, char *path)
 {
-    ListFilesRequest request;
+    /* Construct and send the request to server */
-    ListFilesResponse response;
+    {
-    pb_istream_t input = pb_istream_from_socket(fd);
+        ListFilesRequest request = {};
         pb_ostream_t output = pb_ostream_from_socket(fd);
         uint8_t zero = 0;
         
+        /* In our protocol, path is optional. If it is not given,
+         * the server will list the root directory. */
         if (path == NULL)
         {
             request.has_path = false;
@@ -59,15 +69,25 @@ bool listdir(int fd, char *path)
             strcpy(request.path, path);
         }
         
+        /* Encode the request. It is written to the socket immediately
+         * through our custom stream. */
         if (!pb_encode(&output, ListFilesRequest_fields, &request))
         {
-        fprintf(stderr, "Encoding failed.\n");
+            fprintf(stderr, "Encoding failed: %s\n", PB_GET_ERROR(&output));
             return false;
         }
         
         /* We signal the end of request with a 0 tag. */
         pb_write(&output, &zero, 1);
+    }
     
+    /* Read back the response from server */
+    {
+        ListFilesResponse response = {};
+        pb_istream_t input = pb_istream_from_socket(fd);
+        
+        /* Give a pointer to our callback function, which will handle the
+         * filenames as they arrive. */
         response.file.funcs.decode = &printfile_callback;
         
         if (!pb_decode(&input, ListFilesResponse_fields, &response))
@@ -76,11 +96,14 @@ bool listdir(int fd, char *path)
             return false;
         }
         
+        /* If the message from server decodes properly, but directory was
+         * not found on server side, we get path_error == true. */
         if (response.path_error)
         {
             fprintf(stderr, "Server reported error.\n");
             return false;
         }
+    }
     
     return true;
 }
@@ -96,6 +119,7 @@ int main(int argc, char **argv)
     
     sockfd = socket(AF_INET, SOCK_STREAM, 0);
     
+    /* Connect to server running on localhost:1234 */
     memset(&servaddr, 0, sizeof(servaddr));
     servaddr.sin_family = AF_INET;
     servaddr.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
@@ -107,9 +131,11 @@ int main(int argc, char **argv)
         return 1;
     }
     
+    /* Send the directory listing request */
     if (!listdir(sockfd, path))
         return 2;
     
+    /* Close connection */
     close(sockfd);
     
     return 0;

examples/network_server/server.c

@@ -23,11 +23,16 @@
 #include "fileproto.pb.h"
 #include "common.h"
 
+/* This callback function will be called once during the encoding.
+ * It will write out any number of FileInfo entries, without consuming unnecessary memory.
+ * This is accomplished by fetching the filenames one at a time and encoding them
+ * immediately.
+ */
 bool listdir_callback(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
 {
     DIR *dir = (DIR*) *arg;
     struct dirent *file;
-    FileInfo fileinfo;
+    FileInfo fileinfo = {};
     
     while ((file = readdir(dir)) != NULL)
     {
@@ -35,9 +40,12 @@ bool listdir_callback(pb_ostream_t *stream, const pb_field_t *field, void * cons
         strncpy(fileinfo.name, file->d_name, sizeof(fileinfo.name));
         fileinfo.name[sizeof(fileinfo.name) - 1] = '\0';
         
+        /* This encodes the header for the field, based on the constant info
+         * from pb_field_t. */
         if (!pb_encode_tag_for_field(stream, field))
             return false;
         
+        /* This encodes the data for the field, based on our FileInfo structure. */
         if (!pb_encode_submessage(stream, FileInfo_fields, &fileinfo))
             return false;
     }
@@ -45,13 +53,18 @@ bool listdir_callback(pb_ostream_t *stream, const pb_field_t *field, void * cons
     return true;
 }
 
+/* Handle one arriving client connection.
+ * Clients are expected to send a ListFilesRequest, terminated by a '0'.
+ * Server will respond with a ListFilesResponse message.
+ */
 void handle_connection(int connfd)
 {
-    ListFilesRequest request;
+    DIR *directory = NULL;
-    ListFilesResponse response;
+    
+    /* Decode the message from the client and open the requested directory. */
+    {
+        ListFilesRequest request = {};
         pb_istream_t input = pb_istream_from_socket(connfd);
-    pb_ostream_t output = pb_ostream_from_socket(connfd);
-    DIR *directory;
         
         if (!pb_decode(&input, ListFilesRequest_fields, &request))
         {
@@ -60,19 +73,26 @@ void handle_connection(int connfd)
         }
         
         directory = opendir(request.path);
-    
         printf("Listing directory: %s\n", request.path);
+    }
+    
+    /* List the files in the directory and transmit the response to client */
+    {
+        ListFilesResponse response = {};
+        pb_ostream_t output = pb_ostream_from_socket(connfd);
         
         if (directory == NULL)
         {
             perror("opendir");
             
+            /* Directory was not found, transmit error status */
             response.has_path_error = true;
             response.path_error = true;
             response.file.funcs.encode = NULL;
         }
         else
         {
+            /* Directory was found, transmit filenames */
             response.has_path_error = false;
             response.file.funcs.encode = &listdir_callback;
             response.file.arg = directory;
@@ -80,8 +100,12 @@ void handle_connection(int connfd)
         
         if (!pb_encode(&output, ListFilesResponse_fields, &response))
         {
-        printf("Encoding failed.\n");
+            printf("Encoding failed: %s\n", PB_GET_ERROR(&output));
         }
+    }
+    
+    if (directory != NULL)
+        closedir(directory);
 }
 
 int main(int argc, char **argv)
@@ -90,8 +114,8 @@ int main(int argc, char **argv)
     struct sockaddr_in servaddr;
     int reuse = 1;
     
+    /* Listen on localhost:1234 for TCP connections */
     listenfd = socket(AF_INET, SOCK_STREAM, 0);
-    
     setsockopt(listenfd, SOL_SOCKET, SO_REUSEADDR, &reuse, sizeof(reuse));
     
     memset(&servaddr, 0, sizeof(servaddr));
@@ -112,6 +136,7 @@ int main(int argc, char **argv)
     
     for(;;)
     {
+        /* Wait for a client */
         connfd = accept(listenfd, NULL, NULL);
         
         if (connfd < 0)
@@ -128,4 +153,6 @@ int main(int argc, char **argv)
         
         close(connfd);
     }
+    
+    return 0;
 }

examples/simple/Makefile

@@ -10,6 +10,7 @@ CSRC  = simple.c                   # The main program
 CSRC += simple.pb.c                # The compiled protocol definition
 CSRC += $(NANOPB_DIR)/pb_encode.c  # The nanopb encoder
 CSRC += $(NANOPB_DIR)/pb_decode.c  # The nanopb decoder
+CSRC += $(NANOPB_DIR)/pb_common.c  # The nanopb common parts
 
 # Build rule for the main program
 simple: $(CSRC)

extra/FindNanopb.cmake

@@ -182,8 +182,8 @@ mark_as_advanced(NANOPB_INCLUDE_DIRS)
 # Find nanopb source files
 set(NANOPB_SRCS)
 set(NANOPB_HDRS)
-list(APPEND _nanopb_srcs pb_decode.c pb_encode.c)
+list(APPEND _nanopb_srcs pb_decode.c pb_encode.c pb_common.c)
-list(APPEND _nanopb_hdrs pb_decode.h pb_encode.h pb.h)
+list(APPEND _nanopb_hdrs pb_decode.h pb_encode.h pb_common.h pb.h)
 
 foreach(FIL ${_nanopb_srcs})
   find_file(${FIL}__nano_pb_file NAMES ${FIL} PATHS ${NANOPB_SRC_ROOT_FOLDER} ${NANOPB_INCLUDE_DIRS})

extra/nanopb.mk

@@ -5,7 +5,7 @@
 NANOPB_DIR := $(abspath $(dir $(lastword $(MAKEFILE_LIST)))../)
 
 # Files for the nanopb core
-NANOPB_CORE = $(NANOPB_DIR)/pb_encode.c $(NANOPB_DIR)/pb_decode.c
+NANOPB_CORE = $(NANOPB_DIR)/pb_encode.c $(NANOPB_DIR)/pb_decode.c $(NANOPB_DIR)/pb_common.c
 
 # Check if we are running on Windows
 ifdef windir

extra/pb_syshdr.h

@@ -53,6 +53,16 @@ typedef int bool;
 
 #endif
 
+/* stdlib.h subset */
+#ifdef PB_ENABLE_MALLOC
+#ifdef HAVE_STDLIB_H
+#include <stdlib.h>
+#else
+void *realloc(void *ptr, size_t size);
+void free(void *ptr);
+#endif
+#endif
+
 /* string.h subset */
 #ifdef HAVE_STRING_H
 #include <string.h>

generator/nanopb_generator.py

@@ -1,7 +1,7 @@
 #!/usr/bin/python
 
 '''Generate header file for nanopb from a ProtoBuf FileDescriptorSet.'''
-nanopb_version = "nanopb-0.2.7-dev"
+nanopb_version = "nanopb-0.3.0"
 
 import sys
 
@@ -261,7 +261,7 @@ class Field:
         result = ''
         if self.allocation == 'POINTER':
             if self.rules == 'REPEATED':
-                result += '    size_t ' + self.name + '_count;\n'
+                result += '    pb_size_t ' + self.name + '_count;\n'
             
             if self.pbtype == 'MESSAGE':
                 # Use struct definition, so recursive submessages are possible
@@ -277,43 +277,49 @@ class Field:
             if self.rules == 'OPTIONAL' and self.allocation == 'STATIC':
                 result += '    bool has_' + self.name + ';\n'
             elif self.rules == 'REPEATED' and self.allocation == 'STATIC':
-                result += '    size_t ' + self.name + '_count;\n'
+                result += '    pb_size_t ' + self.name + '_count;\n'
             result += '    %s %s%s;' % (self.ctype, self.name, self.array_decl)
         return result
     
     def types(self):
         '''Return definitions for any special types this field might need.'''
         if self.pbtype == 'BYTES' and self.allocation == 'STATIC':
-            result = 'typedef struct {\n'
+            result = 'typedef PB_BYTES_ARRAY_T(%d) %s;\n' % (self.max_size, self.ctype)
-            result += '    size_t size;\n'
-            result += '    uint8_t bytes[%d];\n' % self.max_size
-            result += '} %s;\n' % self.ctype
         else:
             result = None
         return result
     
-    def default_decl(self, declaration_only = False):
+    def get_initializer(self, null_init):
-        '''Return definition for this field's default value.'''
+        '''Return literal expression for this field's default value.'''
-        if self.default is None:
-            return None
         
-        ctype, default = self.ctype, self.default
+        if self.pbtype == 'MESSAGE':
-        array_decl = ''
+            if null_init:
+                return '%s_init_zero' % self.ctype
+            else:
+                return '%s_init_default' % self.ctype
+        
+        if self.default is None or null_init:
+            if self.pbtype == 'STRING':
+                return '""'
+            elif self.pbtype == 'BYTES':
+                return '{0, {0}}'
+            elif self.pbtype == 'ENUM':
+                return '(%s)0' % self.ctype
+            else:
+                return '0'
+        
+        default = str(self.default)
         
         if self.pbtype == 'STRING':
-            if self.allocation != 'STATIC':
+            default = default.encode('utf-8').encode('string_escape')
-                return None # Not implemented
-        
-            array_decl = '[%d]' % self.max_size
-            default = str(self.default).encode('string_escape')
             default = default.replace('"', '\\"')
             default = '"' + default + '"'
         elif self.pbtype == 'BYTES':
-            if self.allocation != 'STATIC':
+            data = default.decode('string_escape')
-                return None # Not implemented
-
-            data = self.default.decode('string_escape')
             data = ['0x%02x' % ord(c) for c in data]
+            if len(data) == 0:
+                default = '{0, {0}}'
+            else:
                 default = '{%d, {%s}}' % (len(data), ','.join(data))
         elif self.pbtype in ['FIXED32', 'UINT32']:
             default += 'u'
@@ -322,6 +328,25 @@ class Field:
         elif self.pbtype in ['SFIXED64', 'INT64']:
             default += 'll'
         
+        return default
+    
+    def default_decl(self, declaration_only = False):
+        '''Return definition for this field's default value.'''
+        if self.default is None:
+            return None
+
+        ctype = self.ctype
+        default = self.get_initializer(False)
+        array_decl = ''
+        
+        if self.pbtype == 'STRING':
+            if self.allocation != 'STATIC':
+                return None # Not implemented
+            array_decl = '[%d]' % self.max_size
+        elif self.pbtype == 'BYTES':
+            if self.allocation != 'STATIC':
+                return None # Not implemented
+        
         if declaration_only:
             return 'extern const %s %s_default%s;' % (ctype, self.struct_name + self.name, array_decl)
         else:
@@ -336,7 +361,7 @@ class Field:
         '''Return the pb_field_t initializer to use in the constant array.
         prev_field_name is the name of the previous field or None.
         '''
-        result  = '    PB_FIELD2(%3d, ' % self.tag
+        result  = '    PB_FIELD(%3d, ' % self.tag
         result += '%-8s, ' % self.pbtype
         result += '%s, ' % self.rules
         result += '%-8s, ' % self.allocation
@@ -351,6 +376,8 @@ class Field:
             result += '0)'
         elif self.pbtype in ['BYTES', 'STRING'] and self.allocation != 'STATIC':
             result += '0)' # Arbitrary size default values not implemented
+        elif self.rules == 'OPTEXT':
+            result += '0)' # Default value for extensions is not implemented
         else:
             result += '&%s_default)' % (self.struct_name + self.name)
         
@@ -551,6 +578,32 @@ class Message:
                 result += types + '\n'
         return result
     
+    def get_initializer(self, null_init):
+        if not self.ordered_fields:
+            return '{0}'
+    
+        parts = []
+        for field in self.ordered_fields:
+            if field.allocation == 'STATIC':
+                if field.rules == 'REPEATED':
+                    parts.append('0')
+                    parts.append('{'
+                                 + ', '.join([field.get_initializer(null_init)] * field.max_count)
+                                 + '}')
+                elif field.rules == 'OPTIONAL':
+                    parts.append('false')
+                    parts.append(field.get_initializer(null_init))
+                else:
+                    parts.append(field.get_initializer(null_init))
+            elif field.allocation == 'POINTER':
+                parts.append('NULL')
+            elif field.allocation == 'CALLBACK':
+                if field.pbtype == 'EXTENSION':
+                    parts.append('NULL')
+                else:
+                    parts.append('{{NULL}, NULL}')
+        return '{' + ', '.join(parts) + '}'
+    
     def default_decl(self, declaration_only = False):
         result = ""
         for field in self.fields:
@@ -637,13 +690,17 @@ def parse_file(fdesc, file_options):
     
     for names, message in iterate_messages(fdesc, base_name):
         message_options = get_nanopb_suboptions(message, file_options, names)
+        
+        if message_options.skip_message:
+            continue
+        
         messages.append(Message(names, message, message_options))
         for enum in message.enum_type:
             enum_options = get_nanopb_suboptions(enum, message_options, names + enum.name)
             enums.append(Enum(names, enum, enum_options))
     
     for names, extension in iterate_extensions(fdesc, base_name):
-        field_options = get_nanopb_suboptions(extension, file_options, names)
+        field_options = get_nanopb_suboptions(extension, file_options, names + extension.name)
         if field_options.type != nanopb_pb2.FT_IGNORE:
             extensions.append(ExtensionField(names, extension, field_options))
     
@@ -705,11 +762,14 @@ def generate_header(dependencies, headername, enums, messages, extensions, optio
     '''
     
     yield '/* Automatically generated nanopb header */\n'
+    if options.notimestamp:
+        yield '/* Generated by %s */\n\n' % (nanopb_version)
+    else:
         yield '/* Generated by %s at %s. */\n\n' % (nanopb_version, time.asctime())
     
     symbol = make_identifier(headername)
-    yield '#ifndef _PB_%s_\n' % symbol
+    yield '#ifndef PB_%s_INCLUDED\n' % symbol
-    yield '#define _PB_%s_\n' % symbol
+    yield '#define PB_%s_INCLUDED\n' % symbol
     try:
         yield options.libformat % ('pb.h')
     except TypeError:
@@ -719,7 +779,12 @@ def generate_header(dependencies, headername, enums, messages, extensions, optio
     
     for dependency in dependencies:
         noext = os.path.splitext(dependency)[0]
-        yield options.genformat % (noext + '.' + options.extension + '.h')
+        yield options.genformat % (noext + options.extension + '.h')
+        yield '\n'
+
+    yield '#if PB_PROTO_HEADER_VERSION != 30\n'
+    yield '#error Regenerate this file with the current version of nanopb generator.\n'
+    yield '#endif\n'
     yield '\n'
 
     yield '#ifdef __cplusplus\n'
@@ -746,6 +811,15 @@ def generate_header(dependencies, headername, enums, messages, extensions, optio
         yield msg.default_decl(True)
     yield '\n'
     
+    yield '/* Initializer values for message structs */\n'
+    for msg in messages:
+        identifier = '%s_init_default' % msg.name
+        yield '#define %-40s %s\n' % (identifier, msg.get_initializer(False))
+    for msg in messages:
+        identifier = '%s_init_zero' % msg.name
+        yield '#define %-40s %s\n' % (identifier, msg.get_initializer(True))
+    yield '\n'
+    
     yield '/* Field tags (for use in manual encoding/decoding) */\n'
     for msg in sort_dependencies(messages):
         for field in msg.fields:
@@ -778,10 +852,18 @@ def generate_source(headername, enums, messages, extensions, options):
     '''Generate content for a source file.'''
     
     yield '/* Automatically generated nanopb constant definitions */\n'
+    if options.notimestamp:
+        yield '/* Generated by %s */\n\n' % (nanopb_version)
+    else:
         yield '/* Generated by %s at %s. */\n\n' % (nanopb_version, time.asctime())
     yield options.genformat % (headername)
     yield '\n'
     
+    yield '#if PB_PROTO_HEADER_VERSION != 30\n'
+    yield '#error Regenerate this file with the current version of nanopb generator.\n'
+    yield '#endif\n'
+    yield '\n'
+    
     for msg in messages:
         yield msg.default_decl(False)
     
@@ -836,7 +918,7 @@ def generate_source(headername, enums, messages, extensions, options):
                 yield ' * numbers or field sizes that are larger than what can fit in 8 or 16 bit\n'
                 yield ' * field descriptors.\n'
                 yield ' */\n'
-                yield 'STATIC_ASSERT((%s), YOU_MUST_DEFINE_PB_FIELD_32BIT_FOR_MESSAGES_%s)\n'%(assertion,msgs)
+                yield 'PB_STATIC_ASSERT((%s), YOU_MUST_DEFINE_PB_FIELD_32BIT_FOR_MESSAGES_%s)\n'%(assertion,msgs)
             yield '#endif\n\n'
         
         if worst < 65536:
@@ -853,7 +935,7 @@ def generate_source(headername, enums, messages, extensions, options):
                 yield ' * numbers or field sizes that are larger than what can fit in the default\n'
                 yield ' * 8 bit descriptors.\n'
                 yield ' */\n'
-                yield 'STATIC_ASSERT((%s), YOU_MUST_DEFINE_PB_FIELD_16BIT_FOR_MESSAGES_%s)\n'%(assertion,msgs)
+                yield 'PB_STATIC_ASSERT((%s), YOU_MUST_DEFINE_PB_FIELD_16BIT_FOR_MESSAGES_%s)\n'%(assertion,msgs)
             yield '#endif\n\n'
     
     # Add check for sizeof(double)
@@ -869,7 +951,7 @@ def generate_source(headername, enums, messages, extensions, options):
         yield ' * These are not directly supported by nanopb, but see example_avr_double.\n'
         yield ' * To get rid of this error, remove any double fields from your .proto.\n'
         yield ' */\n'
-        yield 'STATIC_ASSERT(sizeof(double) == 8, DOUBLE_MUST_BE_8_BYTES)\n'
+        yield 'PB_STATIC_ASSERT(sizeof(double) == 8, DOUBLE_MUST_BE_8_BYTES)\n'
     
     yield '\n'
 
@@ -951,8 +1033,8 @@ optparser = OptionParser(
              "Output will be written to file.pb.h and file.pb.c.")
 optparser.add_option("-x", dest="exclude", metavar="FILE", action="append", default=[],
     help="Exclude file from generated #include list.")
-optparser.add_option("-e", "--extension", dest="extension", metavar="EXTENSION", default="pb",
+optparser.add_option("-e", "--extension", dest="extension", metavar="EXTENSION", default=".pb",
-    help="Set extension to use instead of 'pb' for generated files. [default: %default]")
+    help="Set extension to use instead of '.pb' for generated files. [default: %default]")
 optparser.add_option("-f", "--options-file", dest="options_file", metavar="FILE", default="%s.options",
     help="Set name of a separate generator options file.")
 optparser.add_option("-Q", "--generated-include-format", dest="genformat",
@@ -961,6 +1043,8 @@ optparser.add_option("-Q", "--generated-include-format", dest="genformat",
 optparser.add_option("-L", "--library-include-format", dest="libformat",
     metavar="FORMAT", default='#include <%s>\n',
     help="Set format string to use for including the nanopb pb.h header. [default: %default]")
+optparser.add_option("-T", "--no-timestamp", dest="notimestamp", action="store_true", default=False,
+    help="Don't add timestamp to .pb.h and .pb.c preambles")
 optparser.add_option("-q", "--quiet", dest="quiet", action="store_true", default=False,
     help="Don't print anything except errors.")
 optparser.add_option("-v", "--verbose", dest="verbose", action="store_true", default=False,
@@ -990,11 +1074,13 @@ def process_file(filename, fdesc, options):
         fdesc = descriptor.FileDescriptorSet.FromString(data).file[0]
     
     # Check if there is a separate .options file
+    had_abspath = False
     try:
         optfilename = options.options_file % os.path.splitext(filename)[0]
     except TypeError:
         # No %s specified, use the filename as-is
         optfilename = options.options_file
+        had_abspath = True
 
     if os.path.isfile(optfilename):
         if options.verbose:
@@ -1002,7 +1088,14 @@ def process_file(filename, fdesc, options):
 
         Globals.separate_options = read_options_file(open(optfilename, "rU"))
     else:
+        # If we are given a full filename and it does not exist, give an error.
+        # However, don't give error when we automatically look for .options file
+        # with the same name as .proto.
+        if options.verbose or had_abspath:
+            sys.stderr.write('Options file not found: ' + optfilename)
+
         Globals.separate_options = []
+
     Globals.matched_namemasks = set()
     
     # Parse the file
@@ -1011,8 +1104,8 @@ def process_file(filename, fdesc, options):
 
     # Decide the file names
     noext = os.path.splitext(filename)[0]
-    headername = noext + '.' + options.extension + '.h'
+    headername = noext + options.extension + '.h'
-    sourcename = noext + '.' + options.extension + '.c'
+    sourcename = noext + options.extension + '.c'
     headerbasename = os.path.basename(headername)
     
     # List of .proto files that should not be included in the C header file

generator/proto/nanopb.proto

@@ -37,6 +37,9 @@ message NanoPBOptions {
   // Note: this cannot be used on CPUs that break on unaligned
   // accesses to variables.
   optional bool packed_struct = 5 [default = false];
+  
+  // Skip this message
+  optional bool skip_message = 6 [default = false];
 }
 
 // Extensions to protoc 'Descriptor' type in order to define options

pb.h

@@ -2,8 +2,8 @@
  * stuff. For the high-level interface, see pb_encode.h and pb_decode.h.
  */
 
-#ifndef _PB_H_
+#ifndef PB_H_INCLUDED
-#define _PB_H_
+#define PB_H_INCLUDED
 
 /*****************************************************************
  * Nanopb compilation time options. You can change these here by *
@@ -46,7 +46,7 @@
 
 /* Version of the nanopb library. Just in case you want to check it in
  * your own program. */
-#define NANOPB_VERSION nanopb-0.2.7-dev
+#define NANOPB_VERSION nanopb-0.3.0
 
 /* Include all the system headers needed by nanopb. You will need the
  * definitions of the following:
@@ -80,8 +80,8 @@
 #   define PB_PACKED_STRUCT_START
 #   define PB_PACKED_STRUCT_END
 #   define pb_packed __attribute__((packed))
-#elif defined(__ICCARM__)
+#elif defined(__ICCARM__) || defined(__CC_ARM)
-    /* For IAR ARM compiler */
+    /* For IAR ARM and Keil MDK-ARM compilers */
 #   define PB_PACKED_STRUCT_START _Pragma("pack(push, 1)")
 #   define PB_PACKED_STRUCT_END _Pragma("pack(pop)")
 #   define pb_packed
@@ -98,23 +98,27 @@
 #endif
 
 /* Handly macro for suppressing unreferenced-parameter compiler warnings. */
-#ifndef UNUSED
+#ifndef PB_UNUSED
-#define UNUSED(x) (void)(x)
+#define PB_UNUSED(x) (void)(x)
 #endif
 
 /* Compile-time assertion, used for checking compatible compilation options.
- * If this does not work properly on your compiler, use #define STATIC_ASSERT
+ * If this does not work properly on your compiler, use
- * to disable it.
+ * #define PB_NO_STATIC_ASSERT to disable it.
  *
  * But before doing that, check carefully the error message / place where it
  * comes from to see if the error has a real cause. Unfortunately the error
  * message is not always very clear to read, but you can see the reason better
- * in the place where the STATIC_ASSERT macro was called.
+ * in the place where the PB_STATIC_ASSERT macro was called.
  */
-#ifndef STATIC_ASSERT
+#ifndef PB_NO_STATIC_ASSERT
-#define STATIC_ASSERT(COND,MSG) typedef char STATIC_ASSERT_MSG(MSG, __LINE__, __COUNTER__)[(COND)?1:-1];
+#ifndef PB_STATIC_ASSERT
-#define STATIC_ASSERT_MSG(MSG, LINE, COUNTER) STATIC_ASSERT_MSG_(MSG, LINE, COUNTER)
+#define PB_STATIC_ASSERT(COND,MSG) typedef char PB_STATIC_ASSERT_MSG(MSG, __LINE__, __COUNTER__)[(COND)?1:-1];
-#define STATIC_ASSERT_MSG_(MSG, LINE, COUNTER) static_assertion_##MSG##LINE##COUNTER
+#define PB_STATIC_ASSERT_MSG(MSG, LINE, COUNTER) PB_STATIC_ASSERT_MSG_(MSG, LINE, COUNTER)
+#define PB_STATIC_ASSERT_MSG_(MSG, LINE, COUNTER) pb_static_assertion_##MSG##LINE##COUNTER
+#endif
+#else
+#define PB_STATIC_ASSERT(COND,MSG)
 #endif
 
 /* Number of required fields to keep track of. */
@@ -187,12 +191,15 @@ typedef uint8_t pb_type_t;
  * and array counts.
  */
 #if defined(PB_FIELD_32BIT)
+#define PB_SIZE_MAX ((uint32_t)-1)
     typedef uint32_t pb_size_t;
     typedef int32_t pb_ssize_t;
 #elif defined(PB_FIELD_16BIT)
+#define PB_SIZE_MAX ((uint16_t)-1)
     typedef uint16_t pb_size_t;
     typedef int16_t pb_ssize_t;
 #else
+#define PB_SIZE_MAX ((uint8_t)-1)
     typedef uint8_t pb_size_t;
     typedef int8_t pb_ssize_t;
 #endif
@@ -206,8 +213,8 @@ typedef uint8_t pb_type_t;
  * PB_FIELD_32BIT.
  */
 PB_PACKED_STRUCT_START
-typedef struct _pb_field_t pb_field_t;
+typedef struct pb_field_s pb_field_t;
-struct _pb_field_t {
+struct pb_field_s {
     pb_size_t tag;
     pb_type_t type;
     pb_size_t data_offset; /* Offset of field data, relative to previous field. */
@@ -228,27 +235,27 @@ PB_PACKED_STRUCT_END
  * If you get errors here, it probably means that your stdint.h is not
  * correct for your platform.
  */
-STATIC_ASSERT(sizeof(int8_t) == 1, INT8_T_WRONG_SIZE)
+PB_STATIC_ASSERT(sizeof(int8_t) == 1, INT8_T_WRONG_SIZE)
-STATIC_ASSERT(sizeof(uint8_t) == 1, UINT8_T_WRONG_SIZE)
+PB_STATIC_ASSERT(sizeof(uint8_t) == 1, UINT8_T_WRONG_SIZE)
-STATIC_ASSERT(sizeof(int16_t) == 2, INT16_T_WRONG_SIZE)
+PB_STATIC_ASSERT(sizeof(int16_t) == 2, INT16_T_WRONG_SIZE)
-STATIC_ASSERT(sizeof(uint16_t) == 2, UINT16_T_WRONG_SIZE)
+PB_STATIC_ASSERT(sizeof(uint16_t) == 2, UINT16_T_WRONG_SIZE)
-STATIC_ASSERT(sizeof(int32_t) == 4, INT32_T_WRONG_SIZE)
+PB_STATIC_ASSERT(sizeof(int32_t) == 4, INT32_T_WRONG_SIZE)
-STATIC_ASSERT(sizeof(uint32_t) == 4, UINT32_T_WRONG_SIZE)
+PB_STATIC_ASSERT(sizeof(uint32_t) == 4, UINT32_T_WRONG_SIZE)
-STATIC_ASSERT(sizeof(int64_t) == 8, INT64_T_WRONG_SIZE)
+PB_STATIC_ASSERT(sizeof(int64_t) == 8, INT64_T_WRONG_SIZE)
-STATIC_ASSERT(sizeof(uint64_t) == 8, UINT64_T_WRONG_SIZE)
+PB_STATIC_ASSERT(sizeof(uint64_t) == 8, UINT64_T_WRONG_SIZE)
 
 /* This structure is used for 'bytes' arrays.
  * It has the number of bytes in the beginning, and after that an array.
  * Note that actual structs used will have a different length of bytes array.
  */
-#define PB_BYTES_ARRAY_T(n) struct { size_t size; uint8_t bytes[n]; }
+#define PB_BYTES_ARRAY_T(n) struct { pb_size_t size; uint8_t bytes[n]; }
 #define PB_BYTES_ARRAY_T_ALLOCSIZE(n) ((size_t)n + offsetof(pb_bytes_array_t, bytes))
 
-struct _pb_bytes_array_t {
+struct pb_bytes_array_s {
-    size_t size;
+    pb_size_t size;
     uint8_t bytes[1];
 };
-typedef struct _pb_bytes_array_t pb_bytes_array_t;
+typedef struct pb_bytes_array_s pb_bytes_array_t;
 
 /* This structure is used for giving the callback function.
  * It is stored in the message structure and filled in by the method that
@@ -268,10 +275,10 @@ typedef struct _pb_bytes_array_t pb_bytes_array_t;
  *
  * The callback can be null if you want to skip a field.
  */
-typedef struct _pb_istream_t pb_istream_t;
+typedef struct pb_istream_s pb_istream_t;
-typedef struct _pb_ostream_t pb_ostream_t;
+typedef struct pb_ostream_s pb_ostream_t;
-typedef struct _pb_callback_t pb_callback_t;
+typedef struct pb_callback_s pb_callback_t;
-struct _pb_callback_t {
+struct pb_callback_s {
 #ifdef PB_OLD_CALLBACK_STYLE
     /* Deprecated since nanopb-0.2.1 */
     union {
@@ -304,9 +311,9 @@ typedef enum {
  * if you want to catch all unknown fields, you can also create a custom
  * pb_extension_type_t with your own callback.
  */
-typedef struct _pb_extension_type_t pb_extension_type_t;
+typedef struct pb_extension_type_s pb_extension_type_t;
-typedef struct _pb_extension_t pb_extension_t;
+typedef struct pb_extension_s pb_extension_t;
-struct _pb_extension_type_t {
+struct pb_extension_type_s {
     /* Called for each unknown field in the message.
      * If you handle the field, read off all of its data and return true.
      * If you do not handle the field, do not read anything and return true.
@@ -328,7 +335,7 @@ struct _pb_extension_type_t {
     const void *arg;
 };
 
-struct _pb_extension_t {
+struct pb_extension_s {
     /* Type describing the extension field. Usually you'll initialize
      * this to a pointer to the automatically generated structure. */
     const pb_extension_type_t *type;
@@ -341,6 +348,10 @@ struct _pb_extension_t {
      * If this extension does not match a field, the next handler is
      * automatically called. */
     pb_extension_t *next;
+
+    /* The decoder sets this to true if the extension was found.
+     * Ignored for encoding. */
+    bool found;
 };
 
 /* Memory allocation functions to use. You can define pb_realloc and
@@ -354,6 +365,9 @@ struct _pb_extension_t {
 #   endif
 #endif
 
+/* This is used to inform about need to regenerate .pb.h/.pb.c files. */
+#define PB_PROTO_HEADER_VERSION 30
+
 /* These macros are used to declare pb_field_t's in the constant array. */
 /* Size of a structure member, in bytes. */
 #define pb_membersize(st, m) (sizeof ((st*)0)->m)
@@ -465,25 +479,14 @@ struct _pb_extension_t {
  *                 SINT32, SINT64, STRING, UINT32, UINT64 or EXTENSION
  * - Field rules:  REQUIRED, OPTIONAL or REPEATED
  * - Allocation:   STATIC or CALLBACK
+ * - Placement: FIRST or OTHER, depending on if this is the first field in structure.
  * - Message name
  * - Field name
  * - Previous field name (or field name again for first field)
  * - Pointer to default value or submsg fields.
  */
 
-#define PB_FIELD(tag, type, rules, allocation, message, field, prevfield, ptr) \
+#define PB_FIELD(tag, type, rules, allocation, placement, message, field, prevfield, ptr) \
-    PB_ ## rules ## _ ## allocation(tag, message, field, \
-        PB_DATAOFFSET_CHOOSE(message, field, prevfield), \
-        PB_LTYPE_MAP_ ## type, ptr)
-
-/* This is a new version of the macro used by nanopb generator from
- * version 0.2.3 onwards. It avoids the use of a ternary expression in
- * the initialization, which confused some compilers.
- *
- * - Placement: FIRST or OTHER, depending on if this is the first field in structure.
- *
- */
-#define PB_FIELD2(tag, type, rules, allocation, placement, message, field, prevfield, ptr) \
         PB_ ## rules ## _ ## allocation(tag, message, field, \
         PB_DATAOFFSET_ ## placement(message, field, prevfield), \
         PB_LTYPE_MAP_ ## type, ptr)
@@ -496,7 +499,11 @@ struct _pb_extension_t {
  * messages if not used.
  */
 #ifdef PB_NO_ERRMSG
-#define PB_RETURN_ERROR(stream,msg) return false
+#define PB_RETURN_ERROR(stream,msg) \
+    do {\
+        PB_UNUSED(stream); \
+        return false; \
+    } while(0)
 #define PB_GET_ERROR(stream) "(errmsg disabled)"
 #else
 #define PB_RETURN_ERROR(stream,msg) \

pb_common.c

@@ -0,0 +1,90 @@
+/* pb_common.c: Common support functions for pb_encode.c and pb_decode.c.
+ *
+ * 2014 Petteri Aimonen <jpa@kapsi.fi>
+ */
+
+#include "pb_common.h"
+
+bool pb_field_iter_begin(pb_field_iter_t *iter, const pb_field_t *fields, void *dest_struct)
+{
+    iter->start = fields;
+    iter->pos = fields;
+    iter->required_field_index = 0;
+    iter->dest_struct = dest_struct;
+    iter->pData = (char*)dest_struct + iter->pos->data_offset;
+    iter->pSize = (char*)iter->pData + iter->pos->size_offset;
+    
+    return (iter->pos->tag != 0);
+}
+
+bool pb_field_iter_next(pb_field_iter_t *iter)
+{
+    const pb_field_t *prev_field = iter->pos;
+
+    if (prev_field->tag == 0)
+    {
+        /* Handle empty message types, where the first field is already the terminator.
+         * In other cases, the iter->pos never points to the terminator. */
+        return false;
+    }
+    
+    iter->pos++;
+    
+    if (iter->pos->tag == 0)
+    {
+        /* Wrapped back to beginning, reinitialize */
+        (void)pb_field_iter_begin(iter, iter->start, iter->dest_struct);
+        return false;
+    }
+    else
+    {
+        /* Increment the pointers based on previous field size */
+        size_t prev_size = prev_field->data_size;
+    
+        if (PB_ATYPE(prev_field->type) == PB_ATYPE_STATIC &&
+            PB_HTYPE(prev_field->type) == PB_HTYPE_REPEATED)
+        {
+            /* In static arrays, the data_size tells the size of a single entry and
+             * array_size is the number of entries */
+            prev_size *= prev_field->array_size;
+        }
+        else if (PB_ATYPE(prev_field->type) == PB_ATYPE_POINTER)
+        {
+            /* Pointer fields always have a constant size in the main structure.
+             * The data_size only applies to the dynamically allocated area. */
+            prev_size = sizeof(void*);
+        }
+        
+        if (PB_HTYPE(prev_field->type) == PB_HTYPE_REQUIRED)
+        {
+            /* Count the required fields, in order to check their presence in the
+             * decoder. */
+            iter->required_field_index++;
+        }
+    
+        iter->pData = (char*)iter->pData + prev_size + iter->pos->data_offset;
+        iter->pSize = (char*)iter->pData + iter->pos->size_offset;
+        return true;
+    }
+}
+
+bool pb_field_iter_find(pb_field_iter_t *iter, uint32_t tag)
+{
+    const pb_field_t *start = iter->pos;
+    
+    do {
+        if (iter->pos->tag == tag &&
+            PB_LTYPE(iter->pos->type) != PB_LTYPE_EXTENSION)
+        {
+            /* Found the wanted field */
+            return true;
+        }
+        
+        (void)pb_field_iter_next(iter);
+    } while (iter->pos != start);
+    
+    /* Searched all the way back to start, and found nothing. */
+    return false;
+}
+
+

pb_common.h

@@ -0,0 +1,42 @@
+/* pb_common.h: Common support functions for pb_encode.c and pb_decode.c.
+ * These functions are rarely needed by applications directly.
+ */
+
+#ifndef PB_COMMON_H_INCLUDED
+#define PB_COMMON_H_INCLUDED
+
+#include "pb.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/* Iterator for pb_field_t list */
+struct pb_field_iter_s {
+    const pb_field_t *start;       /* Start of the pb_field_t array */
+    const pb_field_t *pos;         /* Current position of the iterator */
+    unsigned required_field_index; /* Zero-based index that counts only the required fields */
+    void *dest_struct;             /* Pointer to start of the structure */
+    void *pData;                   /* Pointer to current field value */
+    void *pSize;                   /* Pointer to count/has field */
+};
+typedef struct pb_field_iter_s pb_field_iter_t;
+
+/* Initialize the field iterator structure to beginning.
+ * Returns false if the message type is empty. */
+bool pb_field_iter_begin(pb_field_iter_t *iter, const pb_field_t *fields, void *dest_struct);
+
+/* Advance the iterator to the next field.
+ * Returns false when the iterator wraps back to the first field. */
+bool pb_field_iter_next(pb_field_iter_t *iter);
+
+/* Advance the iterator until it points at a field with the given tag.
+ * Returns false if no such field exists. */
+bool pb_field_iter_find(pb_field_iter_t *iter, uint32_t tag);
+
+#ifdef __cplusplus
+} /* extern "C" */
+#endif
+
+#endif
+

pb_decode.c

@@ -13,39 +13,25 @@
     #define checkreturn __attribute__((warn_unused_result))
 #endif
 
-#define NANOPB_INTERNALS
 #include "pb.h"
 #include "pb_decode.h"
+#include "pb_common.h"
 
 /**************************************
  * Declarations internal to this file *
  **************************************/
 
-/* Iterator for pb_field_t list */
-typedef struct {
-    const pb_field_t *start; /* Start of the pb_field_t array */
-    const pb_field_t *pos; /* Current position of the iterator */
-    unsigned field_index; /* Zero-based index of the field. */
-    unsigned required_field_index; /* Zero-based index that counts only the required fields */
-    void *dest_struct; /* Pointer to the destination structure to decode to */
-    void *pData; /* Pointer where to store current field value */
-    void *pSize; /* Pointer where to store the size of current array field */
-} pb_field_iterator_t;
-
 typedef bool (*pb_decoder_t)(pb_istream_t *stream, const pb_field_t *field, void *dest) checkreturn;
 
 static bool checkreturn buf_read(pb_istream_t *stream, uint8_t *buf, size_t count);
 static bool checkreturn pb_decode_varint32(pb_istream_t *stream, uint32_t *dest);
 static bool checkreturn read_raw_value(pb_istream_t *stream, pb_wire_type_t wire_type, uint8_t *buf, size_t *size);
-static void pb_field_init(pb_field_iterator_t *iter, const pb_field_t *fields, void *dest_struct);
+static bool checkreturn decode_static_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iter_t *iter);
-static bool pb_field_next(pb_field_iterator_t *iter);
+static bool checkreturn decode_callback_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iter_t *iter);
-static bool checkreturn pb_field_find(pb_field_iterator_t *iter, uint32_t tag);
+static bool checkreturn decode_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iter_t *iter);
-static bool checkreturn decode_static_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iterator_t *iter);
-static bool checkreturn decode_callback_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iterator_t *iter);
-static bool checkreturn decode_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iterator_t *iter);
 static bool checkreturn default_extension_decoder(pb_istream_t *stream, pb_extension_t *extension, uint32_t tag, pb_wire_type_t wire_type);
-static bool checkreturn decode_extension(pb_istream_t *stream, uint32_t tag, pb_wire_type_t wire_type, pb_field_iterator_t *iter);
+static bool checkreturn decode_extension(pb_istream_t *stream, uint32_t tag, pb_wire_type_t wire_type, pb_field_iter_t *iter);
-static bool checkreturn find_extension_field(pb_field_iterator_t *iter);
+static bool checkreturn find_extension_field(pb_field_iter_t *iter);
 static void pb_message_set_to_defaults(const pb_field_t fields[], void *dest_struct);
 static bool checkreturn pb_dec_varint(pb_istream_t *stream, const pb_field_t *field, void *dest);
 static bool checkreturn pb_dec_uvarint(pb_istream_t *stream, const pb_field_t *field, void *dest);
@@ -130,7 +116,7 @@ bool checkreturn pb_read(pb_istream_t *stream, uint8_t *buf, size_t count)
  * This is an optimization for the varint decoding. */
 static bool checkreturn pb_readbyte(pb_istream_t *stream, uint8_t *buf)
 {
-    if (!stream->bytes_left)
+    if (stream->bytes_left == 0)
         PB_RETURN_ERROR(stream, "end-of-stream");
 
 #ifndef PB_BUFFER_ONLY
@@ -174,7 +160,7 @@ static bool checkreturn pb_decode_varint32(pb_istream_t *stream, uint32_t *dest)
     if (!pb_readbyte(stream, &byte))
         return false;
     
-    if (!(byte & 0x80))
+    if ((byte & 0x80) == 0)
     {
         /* Quick case, 1 byte value */
         result = byte;
@@ -339,75 +325,11 @@ void pb_close_string_substream(pb_istream_t *stream, pb_istream_t *substream)
 #endif
 }
 
-static void pb_field_init(pb_field_iterator_t *iter, const pb_field_t *fields, void *dest_struct)
-{
-    iter->start = iter->pos = fields;
-    iter->field_index = 0;
-    iter->required_field_index = 0;
-    iter->pData = (char*)dest_struct + iter->pos->data_offset;
-    iter->pSize = (char*)iter->pData + iter->pos->size_offset;
-    iter->dest_struct = dest_struct;
-}
-
-static bool pb_field_next(pb_field_iterator_t *iter)
-{
-    bool notwrapped = true;
-    size_t prev_size = iter->pos->data_size;
-    
-    if (PB_ATYPE(iter->pos->type) == PB_ATYPE_STATIC &&
-        PB_HTYPE(iter->pos->type) == PB_HTYPE_REPEATED)
-    {
-        prev_size *= iter->pos->array_size;
-    }
-    else if (PB_ATYPE(iter->pos->type) == PB_ATYPE_POINTER)
-    {
-        prev_size = sizeof(void*);
-    }
-    
-    if (iter->pos->tag == 0)
-        return false; /* Only happens with empty message types */
-    
-    if (PB_HTYPE(iter->pos->type) == PB_HTYPE_REQUIRED)
-        iter->required_field_index++;
-    
-    iter->pos++;
-    iter->field_index++;
-    if (iter->pos->tag == 0)
-    {
-        iter->pos = iter->start;
-        iter->field_index = 0;
-        iter->required_field_index = 0;
-        iter->pData = iter->dest_struct;
-        prev_size = 0;
-        notwrapped = false;
-    }
-    
-    iter->pData = (char*)iter->pData + prev_size + iter->pos->data_offset;
-    iter->pSize = (char*)iter->pData + iter->pos->size_offset;
-    return notwrapped;
-}
-
-static bool checkreturn pb_field_find(pb_field_iterator_t *iter, uint32_t tag)
-{
-    unsigned start = iter->field_index;
-    
-    do {
-        if (iter->pos->tag == tag &&
-            PB_LTYPE(iter->pos->type) != PB_LTYPE_EXTENSION)
-        {
-            return true;
-        }
-        pb_field_next(iter);
-    } while (iter->field_index != start);
-    
-    return false;
-}
-
 /*************************
  * Decode a single field *
  *************************/
 
-static bool checkreturn decode_static_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iterator_t *iter)
+static bool checkreturn decode_static_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iter_t *iter)
 {
     pb_type_t type;
     pb_decoder_t func;
@@ -430,12 +352,12 @@ static bool checkreturn decode_static_field(pb_istream_t *stream, pb_wire_type_t
             {
                 /* Packed array */
                 bool status = true;
-                size_t *size = (size_t*)iter->pSize;
+                pb_size_t *size = (pb_size_t*)iter->pSize;
                 pb_istream_t substream;
                 if (!pb_make_string_substream(stream, &substream))
                     return false;
                 
-                while (substream.bytes_left && *size < iter->pos->array_size)
+                while (substream.bytes_left > 0 && *size < iter->pos->array_size)
                 {
                     void *pItem = (uint8_t*)iter->pData + iter->pos->data_size * (*size);
                     if (!func(&substream, iter->pos, pItem))
@@ -455,7 +377,7 @@ static bool checkreturn decode_static_field(pb_istream_t *stream, pb_wire_type_t
             else
             {
                 /* Repeated field */
-                size_t *size = (size_t*)iter->pSize;
+                pb_size_t *size = (pb_size_t*)iter->pSize;
                 void *pItem = (uint8_t*)iter->pData + iter->pos->data_size * (*size);
                 if (*size >= iter->pos->array_size)
                     PB_RETURN_ERROR(stream, "array overflow");
@@ -471,16 +393,32 @@ static bool checkreturn decode_static_field(pb_istream_t *stream, pb_wire_type_t
 
 #ifdef PB_ENABLE_MALLOC
 /* Allocate storage for the field and store the pointer at iter->pData.
- * array_size is the number of entries to reserve in an array. */
+ * array_size is the number of entries to reserve in an array.
+ * Zero size is not allowed, use pb_free() for releasing.
+ */
 static bool checkreturn allocate_field(pb_istream_t *stream, void *pData, size_t data_size, size_t array_size)
 {    
     void *ptr = *(void**)pData;
-    size_t size = array_size * data_size;
+    
+    /* Check for multiplication overflows.
+     * This code avoids the costly division if the sizes are small enough.
+     * Multiplication is safe as long as only half of bits are set
+     * in either multiplicand.
+     */
+    const size_t check_limit = (size_t)1 << (sizeof(size_t) * 4);
+    if (data_size >= check_limit || array_size >= check_limit)
+    {
+        const size_t size_max = (size_t)-1;
+        if (size_max / array_size < data_size)
+        {
+            PB_RETURN_ERROR(stream, "size too large");
+        }
+    }
     
     /* Allocate new or expand previous allocation */
     /* Note: on failure the old pointer will remain in the structure,
      * the message must be freed by caller also on error return. */
-    ptr = pb_realloc(ptr, size);
+    ptr = pb_realloc(ptr, array_size * data_size);
     if (ptr == NULL)
         PB_RETURN_ERROR(stream, "realloc failed");
     
@@ -489,7 +427,7 @@ static bool checkreturn allocate_field(pb_istream_t *stream, void *pData, size_t
 }
 
 /* Clear a newly allocated item in case it contains a pointer, or is a submessage. */
-static void initialize_pointer_field(void *pItem, pb_field_iterator_t *iter)
+static void initialize_pointer_field(void *pItem, pb_field_iter_t *iter)
 {
     if (PB_LTYPE(iter->pos->type) == PB_LTYPE_STRING ||
         PB_LTYPE(iter->pos->type) == PB_LTYPE_BYTES)
@@ -503,11 +441,11 @@ static void initialize_pointer_field(void *pItem, pb_field_iterator_t *iter)
 }
 #endif
 
-static bool checkreturn decode_pointer_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iterator_t *iter)
+static bool checkreturn decode_pointer_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iter_t *iter)
 {
 #ifndef PB_ENABLE_MALLOC
-    UNUSED(wire_type);
+    PB_UNUSED(wire_type);
-    UNUSED(iter);
+    PB_UNUSED(iter);
     PB_RETURN_ERROR(stream, "no malloc support");
 #else
     pb_type_t type;
@@ -540,7 +478,7 @@ static bool checkreturn decode_pointer_field(pb_istream_t *stream, pb_wire_type_
             {
                 /* Packed array, multiple items come in at once. */
                 bool status = true;
-                size_t *size = (size_t*)iter->pSize;
+                pb_size_t *size = (pb_size_t*)iter->pSize;
                 size_t allocated_size = *size;
                 void *pItem;
                 pb_istream_t substream;
@@ -550,7 +488,7 @@ static bool checkreturn decode_pointer_field(pb_istream_t *stream, pb_wire_type_
                 
                 while (substream.bytes_left)
                 {
-                    if (*size + 1 > allocated_size)
+                    if ((size_t)*size + 1 > allocated_size)
                     {
                         /* Allocate more storage. This tries to guess the
                          * number of remaining entries. Round the division
@@ -572,6 +510,16 @@ static bool checkreturn decode_pointer_field(pb_istream_t *stream, pb_wire_type_
                         status = false;
                         break;
                     }
+                    
+                    if (*size == PB_SIZE_MAX)
+                    {
+#ifndef PB_NO_ERRMSG
+                        stream->errmsg = "too many array entries";
+#endif
+                        status = false;
+                        break;
+                    }
+                    
                     (*size)++;
                 }
                 pb_close_string_substream(stream, &substream);
@@ -581,9 +529,12 @@ static bool checkreturn decode_pointer_field(pb_istream_t *stream, pb_wire_type_
             else
             {
                 /* Normal repeated field, i.e. only one item at a time. */
-                size_t *size = (size_t*)iter->pSize;
+                pb_size_t *size = (pb_size_t*)iter->pSize;
                 void *pItem;
                 
+                if (*size == PB_SIZE_MAX)
+                    PB_RETURN_ERROR(stream, "too many array entries");
+                
                 (*size)++;
                 if (!allocate_field(stream, iter->pData, iter->pos->data_size, *size))
                     return false;
@@ -599,7 +550,7 @@ static bool checkreturn decode_pointer_field(pb_istream_t *stream, pb_wire_type_
 #endif
 }
 
-static bool checkreturn decode_callback_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iterator_t *iter)
+static bool checkreturn decode_callback_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iter_t *iter)
 {
     pb_callback_t *pCallback = (pb_callback_t*)iter->pData;
     
@@ -646,7 +597,7 @@ static bool checkreturn decode_callback_field(pb_istream_t *stream, pb_wire_type
     }
 }
 
-static bool checkreturn decode_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iterator_t *iter)
+static bool checkreturn decode_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iter_t *iter)
 {
     switch (PB_ATYPE(iter->pos->type))
     {
@@ -670,19 +621,17 @@ static bool checkreturn default_extension_decoder(pb_istream_t *stream,
     pb_extension_t *extension, uint32_t tag, pb_wire_type_t wire_type)
 {
     const pb_field_t *field = (const pb_field_t*)extension->type->arg;
-    pb_field_iterator_t iter;
+    pb_field_iter_t iter;
-    bool dummy;
     
     if (field->tag != tag)
         return true;
     
-    iter.start = field;
+    /* Fake a field iterator for the extension field.
-    iter.pos = field;
+     * It is not actually safe to advance this iterator, but decode_field
-    iter.field_index = 0;
+     * will not even try to. */
-    iter.required_field_index = 0;
+    (void)pb_field_iter_begin(&iter, field, extension->dest);
-    iter.dest_struct = extension->dest;
     iter.pData = extension->dest;
-    iter.pSize = &dummy;
+    iter.pSize = &extension->found;
     
     return decode_field(stream, wire_type, &iter);
 }
@@ -690,12 +639,12 @@ static bool checkreturn default_extension_decoder(pb_istream_t *stream,
 /* Try to decode an unknown field as an extension field. Tries each extension
  * decoder in turn, until one of them handles the field or loop ends. */
 static bool checkreturn decode_extension(pb_istream_t *stream,
-    uint32_t tag, pb_wire_type_t wire_type, pb_field_iterator_t *iter)
+    uint32_t tag, pb_wire_type_t wire_type, pb_field_iter_t *iter)
 {
     pb_extension_t *extension = *(pb_extension_t* const *)iter->pData;
     size_t pos = stream->bytes_left;
     
-    while (extension && pos == stream->bytes_left)
+    while (extension != NULL && pos == stream->bytes_left)
     {
         bool status;
         if (extension->type->decode)
@@ -715,15 +664,15 @@ static bool checkreturn decode_extension(pb_istream_t *stream,
 /* Step through the iterator until an extension field is found or until all
  * entries have been checked. There can be only one extension field per
  * message. Returns false if no extension field is found. */
-static bool checkreturn find_extension_field(pb_field_iterator_t *iter)
+static bool checkreturn find_extension_field(pb_field_iter_t *iter)
 {
-    unsigned start = iter->field_index;
+    const pb_field_t *start = iter->pos;
     
     do {
         if (PB_LTYPE(iter->pos->type) == PB_LTYPE_EXTENSION)
             return true;
-        pb_field_next(iter);
+        (void)pb_field_iter_next(iter);
-    } while (iter->field_index != start);
+    } while (iter->pos != start);
     
     return false;
 }
@@ -731,18 +680,16 @@ static bool checkreturn find_extension_field(pb_field_iterator_t *iter)
 /* Initialize message fields to default values, recursively */
 static void pb_message_set_to_defaults(const pb_field_t fields[], void *dest_struct)
 {
-    pb_field_iterator_t iter;
+    pb_field_iter_t iter;
-    pb_field_init(&iter, fields, dest_struct);
+
+    if (!pb_field_iter_begin(&iter, fields, dest_struct))
+        return; /* Empty message type */
     
     do
     {
         pb_type_t type;
         type = iter.pos->type;
         
-        /* Avoid crash on empty message types (zero fields) */
-        if (iter.pos->tag == 0)
-            continue;
-        
         if (PB_ATYPE(type) == PB_ATYPE_STATIC)
         {
             if (PB_HTYPE(type) == PB_HTYPE_OPTIONAL)
@@ -754,7 +701,7 @@ static void pb_message_set_to_defaults(const pb_field_t fields[], void *dest_str
             else if (PB_HTYPE(type) == PB_HTYPE_REPEATED)
             {
                 /* Set array count to 0, no need to initialize contents. */
-                *(size_t*)iter.pSize = 0;
+                *(pb_size_t*)iter.pSize = 0;
                 continue;
             }
             
@@ -782,14 +729,14 @@ static void pb_message_set_to_defaults(const pb_field_t fields[], void *dest_str
             /* Initialize array count to 0. */
             if (PB_HTYPE(type) == PB_HTYPE_REPEATED)
             {
-                *(size_t*)iter.pSize = 0;
+                *(pb_size_t*)iter.pSize = 0;
             }
         }
         else if (PB_ATYPE(type) == PB_ATYPE_CALLBACK)
         {
             /* Don't overwrite callback */
         }
-    } while (pb_field_next(&iter));
+    } while (pb_field_iter_next(&iter));
 }
 
 /*********************
@@ -798,11 +745,13 @@ static void pb_message_set_to_defaults(const pb_field_t fields[], void *dest_str
 
 bool checkreturn pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct)
 {
-    uint8_t fields_seen[(PB_MAX_REQUIRED_FIELDS + 7) / 8] = {0}; /* Used to check for required fields */
+    uint8_t fields_seen[(PB_MAX_REQUIRED_FIELDS + 7) / 8] = {0, 0, 0, 0, 0, 0, 0, 0};
     uint32_t extension_range_start = 0;
-    pb_field_iterator_t iter;
+    pb_field_iter_t iter;
     
-    pb_field_init(&iter, fields, dest_struct);
+    /* Return value ignored, as empty message types will be correctly handled by
+     * pb_field_iter_find() anyway. */
+    (void)pb_field_iter_begin(&iter, fields, dest_struct);
     
     while (stream->bytes_left)
     {
@@ -818,7 +767,7 @@ bool checkreturn pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[
                 return false;
         }
         
-        if (!pb_field_find(&iter, tag))
+        if (!pb_field_iter_find(&iter, tag))
         {
             /* No match found, check if it matches an extension. */
             if (tag >= extension_range_start)
@@ -871,10 +820,10 @@ bool checkreturn pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[
         do {
             req_field_count = iter.required_field_index;
             last_type = iter.pos->type;
-        } while (pb_field_next(&iter));
+        } while (pb_field_iter_next(&iter));
         
         /* Fixup if last field was also required. */
-        if (PB_HTYPE(last_type) == PB_HTYPE_REQUIRED && iter.pos->tag)
+        if (PB_HTYPE(last_type) == PB_HTYPE_REQUIRED && iter.pos->tag != 0)
             req_field_count++;
         
         /* Check the whole bytes */
@@ -922,18 +871,16 @@ bool pb_decode_delimited(pb_istream_t *stream, const pb_field_t fields[], void *
 #ifdef PB_ENABLE_MALLOC
 void pb_release(const pb_field_t fields[], void *dest_struct)
 {
-    pb_field_iterator_t iter;
+    pb_field_iter_t iter;
-    pb_field_init(&iter, fields, dest_struct);
+    
+    if (!pb_field_iter_begin(&iter, fields, dest_struct))
+        return; /* Empty message type */
     
     do
     {
         pb_type_t type;
         type = iter.pos->type;
     
-        /* Avoid crash on empty message types (zero fields) */
-        if (iter.pos->tag == 0)
-            continue;
-        
         if (PB_ATYPE(type) == PB_ATYPE_POINTER)
         {
             if (PB_HTYPE(type) == PB_HTYPE_REPEATED &&
@@ -942,7 +889,7 @@ void pb_release(const pb_field_t fields[], void *dest_struct)
             {
                 /* Release entries in repeated string or bytes array */
                 void **pItem = *(void***)iter.pData;
-                size_t count = *(size_t*)iter.pSize;
+                pb_size_t count = *(pb_size_t*)iter.pSize;
                 while (count--)
                 {
                     pb_free(*pItem);
@@ -953,11 +900,11 @@ void pb_release(const pb_field_t fields[], void *dest_struct)
             {
                 /* Release fields in submessages */
                 void *pItem = *(void**)iter.pData;
-                size_t count = (pItem ? 1 : 0);
+                pb_size_t count = (pItem ? 1 : 0);
                 
                 if (PB_HTYPE(type) == PB_HTYPE_REPEATED)
                 {
-                    count = *(size_t*)iter.pSize;   
+                    count = *(pb_size_t*)iter.pSize;   
                 }
                 
                 while (count--)
@@ -971,7 +918,7 @@ void pb_release(const pb_field_t fields[], void *dest_struct)
             pb_free(*(void**)iter.pData);
             *(void**)iter.pData = NULL;
         }
-    } while (pb_field_next(&iter));
+    } while (pb_field_iter_next(&iter));
 }
 #endif
 
@@ -1033,7 +980,7 @@ bool pb_decode_fixed64(pb_istream_t *stream, void *dest)
     #endif   
 }
 
-bool checkreturn pb_dec_varint(pb_istream_t *stream, const pb_field_t *field, void *dest)
+static bool checkreturn pb_dec_varint(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
     uint64_t value;
     if (!pb_decode_varint(stream, &value))
@@ -1051,7 +998,7 @@ bool checkreturn pb_dec_varint(pb_istream_t *stream, const pb_field_t *field, vo
     return true;
 }
 
-bool checkreturn pb_dec_uvarint(pb_istream_t *stream, const pb_field_t *field, void *dest)
+static bool checkreturn pb_dec_uvarint(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
     uint64_t value;
     if (!pb_decode_varint(stream, &value))
@@ -1067,7 +1014,7 @@ bool checkreturn pb_dec_uvarint(pb_istream_t *stream, const pb_field_t *field, v
     return true;
 }
 
-bool checkreturn pb_dec_svarint(pb_istream_t *stream, const pb_field_t *field, void *dest)
+static bool checkreturn pb_dec_svarint(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
     int64_t value;
     if (!pb_decode_svarint(stream, &value))
@@ -1083,19 +1030,19 @@ bool checkreturn pb_dec_svarint(pb_istream_t *stream, const pb_field_t *field, v
     return true;
 }
 
-bool checkreturn pb_dec_fixed32(pb_istream_t *stream, const pb_field_t *field, void *dest)
+static bool checkreturn pb_dec_fixed32(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
-    UNUSED(field);
+    PB_UNUSED(field);
     return pb_decode_fixed32(stream, dest);
 }
 
-bool checkreturn pb_dec_fixed64(pb_istream_t *stream, const pb_field_t *field, void *dest)
+static bool checkreturn pb_dec_fixed64(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
-    UNUSED(field);
+    PB_UNUSED(field);
     return pb_decode_fixed64(stream, dest);
 }
 
-bool checkreturn pb_dec_bytes(pb_istream_t *stream, const pb_field_t *field, void *dest)
+static bool checkreturn pb_dec_bytes(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
     uint32_t size;
     pb_bytes_array_t *bdest;
@@ -1120,11 +1067,16 @@ bool checkreturn pb_dec_bytes(pb_istream_t *stream, const pb_field_t *field, voi
         bdest = (pb_bytes_array_t*)dest;
     }
     
-    bdest->size = size;
+    if (size > PB_SIZE_MAX)
+    {
+        PB_RETURN_ERROR(stream, "bytes overflow");
+    }
+
+    bdest->size = (pb_size_t)size;
     return pb_read(stream, bdest->bytes, size);
 }
 
-bool checkreturn pb_dec_string(pb_istream_t *stream, const pb_field_t *field, void *dest)
+static bool checkreturn pb_dec_string(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
     uint32_t size;
     size_t alloc_size;
@@ -1156,7 +1108,7 @@ bool checkreturn pb_dec_string(pb_istream_t *stream, const pb_field_t *field, vo
     return status;
 }
 
-bool checkreturn pb_dec_submessage(pb_istream_t *stream, const pb_field_t *field, void *dest)
+static bool checkreturn pb_dec_submessage(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
     bool status;
     pb_istream_t substream;

pb_decode.h

@@ -3,8 +3,8 @@
  * field descriptions created by nanopb_generator.py.
  */
 
-#ifndef _PB_DECODE_H_
+#ifndef PB_DECODE_H_INCLUDED
-#define _PB_DECODE_H_
+#define PB_DECODE_H_INCLUDED
 
 #include "pb.h"
 
@@ -25,7 +25,7 @@ extern "C" {
  *    is different than from the main stream. Don't use bytes_left to compute
  *    any pointers.
  */
-struct _pb_istream_t
+struct pb_istream_s
 {
 #ifdef PB_BUFFER_ONLY
     /* Callback pointer is not used in buffer-only configuration.

pb_encode.c

@@ -3,9 +3,9 @@
  * 2011 Petteri Aimonen <jpa@kapsi.fi>
  */
 
-#define NANOPB_INTERNALS
 #include "pb.h"
 #include "pb_encode.h"
+#include "pb_common.h"
 
 /* Use the GCC warn_unused_result attribute to check that all return values
  * are propagated correctly. On other compilers and gcc before 3.4.0 just
@@ -246,7 +246,7 @@ static bool checkreturn encode_basic_field(pb_ostream_t *stream,
             break;
         
         case PB_HTYPE_REPEATED:
-            if (!encode_array(stream, field, pData, *(const size_t*)pSize, func))
+            if (!encode_array(stream, field, pData, *(const pb_size_t*)pSize, func))
                 return false;
             break;
         
@@ -311,7 +311,7 @@ static bool checkreturn encode_extension_field(pb_ostream_t *stream,
     const pb_field_t *field, const void *pData)
 {
     const pb_extension_t *extension = *(const pb_extension_t* const *)pData;
-    UNUSED(field);
+    PB_UNUSED(field);
     
     while (extension)
     {
@@ -334,42 +334,38 @@ static bool checkreturn encode_extension_field(pb_ostream_t *stream,
  * Encode all fields *
  *********************/
 
+static void *remove_const(const void *p)
+{
+    /* Note: this casts away const, in order to use the common field iterator
+     * logic for both encoding and decoding. */
+    union {
+        void *p1;
+        const void *p2;
+    } t;
+    t.p2 = p;
+    return t.p1;
+}
+
 bool checkreturn pb_encode(pb_ostream_t *stream, const pb_field_t fields[], const void *src_struct)
 {
-    const pb_field_t *field = fields;
+    pb_field_iter_t iter;
-    const void *pData = src_struct;
+    if (!pb_field_iter_begin(&iter, fields, remove_const(src_struct)))
-    size_t prev_size = 0;
+        return true; /* Empty message type */
     
-    while (field->tag != 0)
+    do {
-    {
+        if (PB_LTYPE(iter.pos->type) == PB_LTYPE_EXTENSION)
-        pData = (const char*)pData + prev_size + field->data_offset;
-        if (PB_ATYPE(field->type) == PB_ATYPE_POINTER)
-            prev_size = sizeof(const void*);
-        else
-            prev_size = field->data_size;
-        
-        /* Special case for static arrays */
-        if (PB_ATYPE(field->type) == PB_ATYPE_STATIC &&
-            PB_HTYPE(field->type) == PB_HTYPE_REPEATED)
-        {
-            prev_size *= field->array_size;
-        }
-        
-        if (PB_LTYPE(field->type) == PB_LTYPE_EXTENSION)
         {
             /* Special case for the extension field placeholder */
-            if (!encode_extension_field(stream, field, pData))
+            if (!encode_extension_field(stream, iter.pos, iter.pData))
                 return false;
         }
         else
         {
             /* Regular field */
-            if (!encode_field(stream, field, pData))
+            if (!encode_field(stream, iter.pos, iter.pData))
                 return false;
         }
-    
+    } while (pb_field_iter_next(&iter));
-        field++;
-    }
     
     return true;
 }
@@ -379,6 +375,17 @@ bool pb_encode_delimited(pb_ostream_t *stream, const pb_field_t fields[], const
     return pb_encode_submessage(stream, fields, src_struct);
 }
 
+bool pb_get_encoded_size(size_t *size, const pb_field_t fields[], const void *src_struct)
+{
+    pb_ostream_t stream = PB_OSTREAM_SIZING;
+    
+    if (!pb_encode(&stream, fields, src_struct))
+        return false;
+    
+    *size = stream.bytes_written;
+    return true;
+}
+
 /********************
  * Helper functions *
  ********************/
@@ -405,9 +412,9 @@ bool checkreturn pb_encode_svarint(pb_ostream_t *stream, int64_t value)
 {
     uint64_t zigzagged;
     if (value < 0)
-        zigzagged = (uint64_t)(~(value << 1));
+        zigzagged = ~((uint64_t)value << 1);
     else
-        zigzagged = (uint64_t)(value << 1);
+        zigzagged = (uint64_t)value << 1;
     
     return pb_encode_varint(stream, zigzagged);
 }
@@ -448,7 +455,7 @@ bool checkreturn pb_encode_fixed64(pb_ostream_t *stream, const void *value)
 
 bool checkreturn pb_encode_tag(pb_ostream_t *stream, pb_wire_type_t wiretype, uint32_t field_number)
 {
-    uint64_t tag = wiretype | (field_number << 3);
+    uint64_t tag = ((uint64_t)field_number << 3) | wiretype;
     return pb_encode_varint(stream, tag);
 }
 
@@ -544,7 +551,7 @@ bool checkreturn pb_encode_submessage(pb_ostream_t *stream, const pb_field_t fie
 
 /* Field encoders */
 
-bool checkreturn pb_enc_varint(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_varint(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
     int64_t value = 0;
     
@@ -562,7 +569,7 @@ bool checkreturn pb_enc_varint(pb_ostream_t *stream, const pb_field_t *field, co
     return pb_encode_varint(stream, (uint64_t)value);
 }
 
-bool checkreturn pb_enc_uvarint(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_uvarint(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
     uint64_t value = 0;
     
@@ -576,7 +583,7 @@ bool checkreturn pb_enc_uvarint(pb_ostream_t *stream, const pb_field_t *field, c
     return pb_encode_varint(stream, value);
 }
 
-bool checkreturn pb_enc_svarint(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_svarint(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
     int64_t value = 0;
     
@@ -590,19 +597,19 @@ bool checkreturn pb_enc_svarint(pb_ostream_t *stream, const pb_field_t *field, c
     return pb_encode_svarint(stream, value);
 }
 
-bool checkreturn pb_enc_fixed64(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_fixed64(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
-    UNUSED(field);
+    PB_UNUSED(field);
     return pb_encode_fixed64(stream, src);
 }
 
-bool checkreturn pb_enc_fixed32(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_fixed32(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
-    UNUSED(field);
+    PB_UNUSED(field);
     return pb_encode_fixed32(stream, src);
 }
 
-bool checkreturn pb_enc_bytes(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_bytes(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
     const pb_bytes_array_t *bytes = (const pb_bytes_array_t*)src;
     
@@ -621,9 +628,8 @@ bool checkreturn pb_enc_bytes(pb_ostream_t *stream, const pb_field_t *field, con
     return pb_encode_string(stream, bytes->bytes, bytes->size);
 }
 
-bool checkreturn pb_enc_string(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_string(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
-    /* strnlen() is not always available, so just use a loop */
     size_t size = 0;
     size_t max_size = field->data_size;
     const char *p = (const char*)src;
@@ -637,6 +643,7 @@ bool checkreturn pb_enc_string(pb_ostream_t *stream, const pb_field_t *field, co
     }
     else
     {
+        /* strnlen() is not always available, so just use a loop */
         while (size < max_size && *p != '\0')
         {
             size++;
@@ -647,7 +654,7 @@ bool checkreturn pb_enc_string(pb_ostream_t *stream, const pb_field_t *field, co
     return pb_encode_string(stream, (const uint8_t*)src, size);
 }
 
-bool checkreturn pb_enc_submessage(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_submessage(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
     if (field->ptr == NULL)
         PB_RETURN_ERROR(stream, "invalid field descriptor");

pb_encode.h

@@ -3,8 +3,8 @@
  * field descriptions created by nanopb_generator.py.
  */
 
-#ifndef _PB_ENCODE_H_
+#ifndef PB_ENCODE_H_INCLUDED
-#define _PB_ENCODE_H_
+#define PB_ENCODE_H_INCLUDED
 
 #include "pb.h"
 
@@ -24,7 +24,7 @@ extern "C" {
  * 4) Substreams will modify max_size and bytes_written. Don't use them
  *    to calculate any pointers.
  */
-struct _pb_ostream_t
+struct pb_ostream_s
 {
 #ifdef PB_BUFFER_ONLY
     /* Callback pointer is not used in buffer-only configuration.
@@ -71,6 +71,10 @@ bool pb_encode(pb_ostream_t *stream, const pb_field_t fields[], const void *src_
  */
 bool pb_encode_delimited(pb_ostream_t *stream, const pb_field_t fields[], const void *src_struct);
 
+/* Encode the message to get the size of the encoded data, but do not store
+ * the data. */
+bool pb_get_encoded_size(size_t *size, const pb_field_t fields[], const void *src_struct);
+
 /**************************************
  * Functions for manipulating streams *
  **************************************/

tests/Makefile

@@ -4,3 +4,18 @@ all:
 clean:
 	scons -c
 
+coverage:
+	rm -rf build coverage
+
+	# LCOV does not like the newer gcov format
+	scons CC=gcc-4.6 CXX=gcc-4.6
+
+	# We are only interested in pb_encode.o and pb_decode.o
+	find build -name '*.gcda' -and \! \( -name '*pb_encode*' -or -name '*pb_decode*' \) -exec rm '{}' \;
+
+	# Collect the data
+	mkdir build/coverage
+	lcov --base-directory . --directory build/ --gcov-tool gcov-4.6 -c -o build/coverage/nanopb.info
+
+	# Generate HTML
+	genhtml -o build/coverage build/coverage/nanopb.info

tests/SConstruct

@@ -19,8 +19,8 @@ env = Environment(ENV = os.environ, tools = ['default', 'nanopb'])
 # Allow overriding the compiler with scons CC=???
 if 'CC' in ARGUMENTS: env.Replace(CC = ARGUMENTS['CC'])
 if 'CXX' in ARGUMENTS: env.Replace(CXX = ARGUMENTS['CXX'])
-if 'CFLAGS' in ARGUMENTS: env.Append(CCFLAGS = ARGUMENTS['CFLAGS'])
+if 'CCFLAGS' in ARGUMENTS: env.Append(CCFLAGS = ARGUMENTS['CCFLAGS'])
-if 'CXXFLAGS' in ARGUMENTS: env.Append(CCFLAGS = ARGUMENTS['CXXFLAGS'])
+if 'CXXFLAGS' in ARGUMENTS: env.Append(CXXFLAGS = ARGUMENTS['CXXFLAGS'])
 
 # Add the builders defined in site_init.py
 add_nanopb_builders(env)
@@ -33,13 +33,16 @@ env.Append(PROTOCPATH = '#../generator')
 
 # Check the compilation environment, unless we are just cleaning up.
 if not env.GetOption('clean'):
-    def check_ccflags(context, flags):
+    def check_ccflags(context, flags, linkflags = ''):
         '''Check if given CCFLAGS are supported'''
         context.Message('Checking support for CCFLAGS="%s"... ' % flags)
         oldflags = context.env['CCFLAGS']
+        oldlinkflags = context.env['CCFLAGS']
         context.env.Append(CCFLAGS = flags)
+        context.env.Append(LINKFLAGS = linkflags)
         result = context.TryCompile("int main() {return 0;}", '.c')
         context.env.Replace(CCFLAGS = oldflags)
+        context.env.Replace(LINKFLAGS = oldlinkflags)
         context.Result(result)
         return result
     
@@ -50,6 +53,7 @@ if not env.GetOption('clean'):
     stdint = conf.CheckCHeader('stdint.h')
     stddef = conf.CheckCHeader('stddef.h')
     string = conf.CheckCHeader('string.h')
+    stdlib = conf.CheckCHeader('stdlib.h')
     if not stdbool or not stdint or not stddef or not string:
         conf.env.Append(CPPDEFINES = {'PB_SYSTEM_HEADER': '\\"pb_syshdr.h\\"'})
         conf.env.Append(CPPPATH = "#../extra")
@@ -58,6 +62,7 @@ if not env.GetOption('clean'):
         if stdint: conf.env.Append(CPPDEFINES = {'HAVE_STDINT_H': 1})
         if stddef: conf.env.Append(CPPDEFINES = {'HAVE_STDDEF_H': 1})
         if string: conf.env.Append(CPPDEFINES = {'HAVE_STRING_H': 1})
+        if stdlib: conf.env.Append(CPPDEFINES = {'HAVE_STDLIB_H': 1})
     
     # Check if we can use pkg-config to find protobuf include path
     status, output = conf.TryAction('pkg-config protobuf --variable=includedir > $TARGET')
@@ -81,6 +86,13 @@ if not env.GetOption('clean'):
         if conf.CheckCCFLAGS(extra):
             conf.env.Append(CORECFLAGS = extra)
     
+    # Check if we can use undefined behaviour sanitizer (only with clang)
+    extra = '-fsanitize=undefined '
+    if 'clang' in env['CC']:
+        if conf.CheckCCFLAGS(extra, linkflags = extra):
+            conf.env.Append(CORECFLAGS = extra)
+            conf.env.Append(LINKFLAGS = extra)
+    
     # End the config stuff
     env = conf.Finish()
 
@@ -89,15 +101,15 @@ if 'gcc' in env['CC']:
     # GNU Compiler Collection
     
     # Debug info, warnings as errors
-    env.Append(CFLAGS = '-ansi -pedantic -g -O0 -Wall -Werror --coverage -fstack-protector-all')
+    env.Append(CFLAGS = '-ansi -pedantic -g -Wall -Werror -fprofile-arcs -ftest-coverage -fstack-protector-all')
     env.Append(CORECFLAGS = '-Wextra')
-    env.Append(LINKFLAGS = '--coverage')
+    env.Append(LINKFLAGS = '-g --coverage')
     
     # We currently need uint64_t anyway, even though ANSI C90 otherwise..
     env.Append(CFLAGS = '-Wno-long-long')
 elif 'clang' in env['CC']:
     # CLang
-    env.Append(CFLAGS = '-ansi -g -O0 -Wall -Werror')
+    env.Append(CFLAGS = '-ansi -g -Wall -Werror')
     env.Append(CORECFLAGS = ' -Wextra -Wcast-qual -Wconversion')
 elif 'cl' in env['CC']:
     # Microsoft Visual C++
@@ -117,10 +129,10 @@ elif 'tcc' in env['CC']:
 
 env.SetDefault(CORECFLAGS = '')
 
-if 'clang++' in env['CXX']:
+if 'clang' in env['CXX']:
-    env.Append(CXXFLAGS = '-g -O0 -Wall -Werror -Wextra -Wno-missing-field-initializers')
+    env.Append(CXXFLAGS = '-g -Wall -Werror -Wextra -Wno-missing-field-initializers')
-elif 'g++' in env['CXX']:
+elif 'g++' in env['CXX'] or 'gcc' in env['CXX']:
-    env.Append(CXXFLAGS = '-g -O0 -Wall -Werror -Wextra -Wno-missing-field-initializers')
+    env.Append(CXXFLAGS = '-g -Wall -Werror -Wextra -Wno-missing-field-initializers')
 elif 'cl' in env['CXX']:
     env.Append(CXXFLAGS = '/Zi /W2 /WX')
     
@@ -129,6 +141,6 @@ import os.path
 env['VARIANT_DIR'] = 'build'
 env['BUILD'] = '#' + env['VARIANT_DIR']
 env['COMMON'] = '#' + env['VARIANT_DIR'] + '/common'
-for subdir in Glob('*/SConscript'):
+for subdir in Glob('*/SConscript') + Glob('regression/*/SConscript'):
     SConscript(subdir, exports = 'env', variant_dir = env['VARIANT_DIR'] + '/' + os.path.dirname(str(subdir)))
 

tests/alltypes/SConscript

@@ -4,8 +4,8 @@
 Import("env")
 
 env.NanopbProto(["alltypes", "alltypes.options"])
-enc = env.Program(["encode_alltypes.c", "alltypes.pb.c", "$COMMON/pb_encode.o"])
+enc = env.Program(["encode_alltypes.c", "alltypes.pb.c", "$COMMON/pb_encode.o", "$COMMON/pb_common.o"])
-dec = env.Program(["decode_alltypes.c", "alltypes.pb.c", "$COMMON/pb_decode.o"])
+dec = env.Program(["decode_alltypes.c", "alltypes.pb.c", "$COMMON/pb_decode.o", "$COMMON/pb_common.o"])
 
 # Test the round-trip from nanopb encoder to nanopb decoder
 env.RunTest(enc)

tests/alltypes/decode_alltypes.c

@@ -19,10 +19,12 @@
    the decoding and checks the fields. */
 bool check_alltypes(pb_istream_t *stream, int mode)
 {
-    AllTypes alltypes;
+    /* Uses _init_default to just make sure that it works. */
+    AllTypes alltypes = AllTypes_init_default;
     
     /* Fill with garbage to better detect initialization errors */
     memset(&alltypes, 0xAA, sizeof(alltypes));
+    alltypes.extensions = 0;
     
     if (!pb_decode(stream, AllTypes_fields, &alltypes))
         return false;

tests/alltypes/encode_alltypes.c

@@ -13,7 +13,7 @@ int main(int argc, char **argv)
     int mode = (argc > 1) ? atoi(argv[1]) : 0;
     
     /* Initialize the structure with constants */
-    AllTypes alltypes = {0};
+    AllTypes alltypes = AllTypes_init_zero;
     
     alltypes.req_int32         = -1001;
     alltypes.req_int64         = -1002;

tests/alltypes_callback/SConscript

@@ -6,8 +6,8 @@ c = Copy("$TARGET", "$SOURCE")
 env.Command("alltypes.proto", "#alltypes/alltypes.proto", c)
 
 env.NanopbProto(["alltypes", "alltypes.options"])
-enc = env.Program(["encode_alltypes_callback.c", "alltypes.pb.c", "$COMMON/pb_encode.o"])
+enc = env.Program(["encode_alltypes_callback.c", "alltypes.pb.c", "$COMMON/pb_encode.o", "$COMMON/pb_common.o"])
-dec = env.Program(["decode_alltypes_callback.c", "alltypes.pb.c", "$COMMON/pb_decode.o"])
+dec = env.Program(["decode_alltypes_callback.c", "alltypes.pb.c", "$COMMON/pb_decode.o", "$COMMON/pb_common.o"])
 
 refdec = "$BUILD/alltypes/decode_alltypes$PROGSUFFIX"
 

tests/alltypes_callback/decode_alltypes_callback.c

@@ -220,6 +220,7 @@ bool check_alltypes(pb_istream_t *stream, int mode)
     
     /* Fill with garbage to better detect initialization errors */
     memset(&alltypes, 0xAA, sizeof(alltypes));
+    alltypes.extensions = 0;
     
     alltypes.req_int32.funcs.decode = &read_varint;
     alltypes.req_int32.arg = (void*)-1001;

tests/alltypes_pointer/SConscript

@@ -18,13 +18,14 @@ strict = env.Clone()
 strict.Append(CFLAGS = strict['CORECFLAGS'])
 strict.Object("pb_decode_with_malloc.o", "$NANOPB/pb_decode.c")
 strict.Object("pb_encode_with_malloc.o", "$NANOPB/pb_encode.c")
+strict.Object("pb_common_with_malloc.o", "$NANOPB/pb_common.c")
 
 c = Copy("$TARGET", "$SOURCE")
 env.Command("alltypes.proto", "#alltypes/alltypes.proto", c)
 
 env.NanopbProto(["alltypes", "alltypes.options"])
-enc = env.Program(["encode_alltypes_pointer.c", "alltypes.pb.c", "pb_encode_with_malloc.o"])
+enc = env.Program(["encode_alltypes_pointer.c", "alltypes.pb.c", "pb_encode_with_malloc.o", "pb_common_with_malloc.o"])
-dec = env.Program(["decode_alltypes_pointer.c", "alltypes.pb.c", "pb_decode_with_malloc.o"])
+dec = env.Program(["decode_alltypes_pointer.c", "alltypes.pb.c", "pb_decode_with_malloc.o", "pb_common_with_malloc.o"])
 
 # Encode and compare results to non-pointer alltypes test case
 env.RunTest(enc)

tests/alltypes_pointer/decode_alltypes_pointer.c

@@ -19,6 +19,7 @@ bool check_alltypes(pb_istream_t *stream, int mode)
     
     /* Fill with garbage to better detect initialization errors */
     memset(&alltypes, 0xAA, sizeof(alltypes));
+    alltypes.extensions = 0;
     
     if (!pb_decode(stream, AllTypes_fields, &alltypes))
         return false;

tests/backwards_compatibility/SConscript

@@ -3,8 +3,8 @@
 
 Import("env")
 
-enc = env.Program(["encode_legacy.c", "alltypes_legacy.c", "$COMMON/pb_encode.o"])
+enc = env.Program(["encode_legacy.c", "alltypes_legacy.c", "$COMMON/pb_encode.o", "$COMMON/pb_common.o"])
-dec = env.Program(["decode_legacy.c", "alltypes_legacy.c", "$COMMON/pb_decode.o"])
+dec = env.Program(["decode_legacy.c", "alltypes_legacy.c", "$COMMON/pb_decode.o", "$COMMON/pb_common.o"])
 
 env.RunTest(enc)
 env.RunTest([dec, "encode_legacy.output"])

tests/backwards_compatibility/alltypes_legacy.c

@@ -1,27 +1,37 @@
 /* Automatically generated nanopb constant definitions */
-/* Generated by 0.2.0-dev at Sun Feb 17 00:09:53 2013. */
+/* Generated by nanopb-0.3.0-dev at Tue Aug 19 17:53:24 2014. */
-/* This is a file generated using nanopb-0.2.0-dev.
- * It is used as a part of test suite in order to detect any
- * incompatible changes made to the generator in future versions.
- */
 
 #include "alltypes_legacy.h"
 
+#if PB_PROTO_HEADER_VERSION != 30
+#error Regenerate this file with the current version of nanopb generator.
+#endif
+
 const char SubMessage_substuff1_default[16] = "1";
 const int32_t SubMessage_substuff2_default = 2;
-const uint32_t SubMessage_substuff3_default = 3;
+const uint32_t SubMessage_substuff3_default = 3u;
+const int32_t Limits_int32_min_default = 2147483647;
+const int32_t Limits_int32_max_default = -2147483647;
+const uint32_t Limits_uint32_min_default = 4294967295u;
+const uint32_t Limits_uint32_max_default = 0u;
+const int64_t Limits_int64_min_default = 9223372036854775807ll;
+const int64_t Limits_int64_max_default = -9223372036854775807ll;
+const uint64_t Limits_uint64_min_default = 18446744073709551615ull;
+const uint64_t Limits_uint64_max_default = 0ull;
+const HugeEnum Limits_enum_min_default = HugeEnum_Positive;
+const HugeEnum Limits_enum_max_default = HugeEnum_Negative;
 const int32_t AllTypes_opt_int32_default = 4041;
-const int64_t AllTypes_opt_int64_default = 4042;
+const int64_t AllTypes_opt_int64_default = 4042ll;
-const uint32_t AllTypes_opt_uint32_default = 4043;
+const uint32_t AllTypes_opt_uint32_default = 4043u;
-const uint64_t AllTypes_opt_uint64_default = 4044;
+const uint64_t AllTypes_opt_uint64_default = 4044ull;
 const int32_t AllTypes_opt_sint32_default = 4045;
 const int64_t AllTypes_opt_sint64_default = 4046;
 const bool AllTypes_opt_bool_default = false;
-const uint32_t AllTypes_opt_fixed32_default = 4048;
+const uint32_t AllTypes_opt_fixed32_default = 4048u;
 const int32_t AllTypes_opt_sfixed32_default = 4049;
 const float AllTypes_opt_float_default = 4050;
-const uint64_t AllTypes_opt_fixed64_default = 4051;
+const uint64_t AllTypes_opt_fixed64_default = 4051ull;
-const int64_t AllTypes_opt_sfixed64_default = 4052;
+const int64_t AllTypes_opt_sfixed64_default = 4052ll;
 const double AllTypes_opt_double_default = 4053;
 const char AllTypes_opt_string_default[16] = "4054";
 const AllTypes_opt_bytes_t AllTypes_opt_bytes_default = {4, {0x34,0x30,0x35,0x35}};
@@ -29,65 +39,115 @@ const MyEnum AllTypes_opt_enum_default = MyEnum_Second;
 
 
 const pb_field_t SubMessage_fields[4] = {
-    PB_FIELD(  1, STRING  , REQUIRED, STATIC, SubMessage, substuff1, substuff1, &SubMessage_substuff1_default),
+    PB_FIELD(  1, STRING  , REQUIRED, STATIC  , FIRST, SubMessage, substuff1, substuff1, &SubMessage_substuff1_default),
-    PB_FIELD(  2, INT32   , REQUIRED, STATIC, SubMessage, substuff2, substuff1, &SubMessage_substuff2_default),
+    PB_FIELD(  2, INT32   , REQUIRED, STATIC  , OTHER, SubMessage, substuff2, substuff1, &SubMessage_substuff2_default),
-    PB_FIELD(  3, FIXED32 , OPTIONAL, STATIC, SubMessage, substuff3, substuff2, &SubMessage_substuff3_default),
+    PB_FIELD(  3, FIXED32 , OPTIONAL, STATIC  , OTHER, SubMessage, substuff3, substuff2, &SubMessage_substuff3_default),
     PB_LAST_FIELD
 };
 
-const pb_field_t AllTypes_fields[53] = {
+const pb_field_t EmptyMessage_fields[1] = {
-    PB_FIELD(  1, INT32   , REQUIRED, STATIC, AllTypes, req_int32, req_int32, 0),
-    PB_FIELD(  2, INT64   , REQUIRED, STATIC, AllTypes, req_int64, req_int32, 0),
-    PB_FIELD(  3, UINT32  , REQUIRED, STATIC, AllTypes, req_uint32, req_int64, 0),
-    PB_FIELD(  4, UINT64  , REQUIRED, STATIC, AllTypes, req_uint64, req_uint32, 0),
-    PB_FIELD(  5, SINT32  , REQUIRED, STATIC, AllTypes, req_sint32, req_uint64, 0),
-    PB_FIELD(  6, SINT64  , REQUIRED, STATIC, AllTypes, req_sint64, req_sint32, 0),
-    PB_FIELD(  7, BOOL    , REQUIRED, STATIC, AllTypes, req_bool, req_sint64, 0),
-    PB_FIELD(  8, FIXED32 , REQUIRED, STATIC, AllTypes, req_fixed32, req_bool, 0),
-    PB_FIELD(  9, SFIXED32, REQUIRED, STATIC, AllTypes, req_sfixed32, req_fixed32, 0),
-    PB_FIELD( 10, FLOAT   , REQUIRED, STATIC, AllTypes, req_float, req_sfixed32, 0),
-    PB_FIELD( 11, FIXED64 , REQUIRED, STATIC, AllTypes, req_fixed64, req_float, 0),
-    PB_FIELD( 12, SFIXED64, REQUIRED, STATIC, AllTypes, req_sfixed64, req_fixed64, 0),
-    PB_FIELD( 13, DOUBLE  , REQUIRED, STATIC, AllTypes, req_double, req_sfixed64, 0),
-    PB_FIELD( 14, STRING  , REQUIRED, STATIC, AllTypes, req_string, req_double, 0),
-    PB_FIELD( 15, BYTES   , REQUIRED, STATIC, AllTypes, req_bytes, req_string, 0),
-    PB_FIELD( 16, MESSAGE , REQUIRED, STATIC, AllTypes, req_submsg, req_bytes, &SubMessage_fields),
-    PB_FIELD( 17, ENUM    , REQUIRED, STATIC, AllTypes, req_enum, req_submsg, 0),
-    PB_FIELD( 21, INT32   , REPEATED, STATIC, AllTypes, rep_int32, req_enum, 0),
-    PB_FIELD( 22, INT64   , REPEATED, STATIC, AllTypes, rep_int64, rep_int32, 0),
-    PB_FIELD( 23, UINT32  , REPEATED, STATIC, AllTypes, rep_uint32, rep_int64, 0),
-    PB_FIELD( 24, UINT64  , REPEATED, STATIC, AllTypes, rep_uint64, rep_uint32, 0),
-    PB_FIELD( 25, SINT32  , REPEATED, STATIC, AllTypes, rep_sint32, rep_uint64, 0),
-    PB_FIELD( 26, SINT64  , REPEATED, STATIC, AllTypes, rep_sint64, rep_sint32, 0),
-    PB_FIELD( 27, BOOL    , REPEATED, STATIC, AllTypes, rep_bool, rep_sint64, 0),
-    PB_FIELD( 28, FIXED32 , REPEATED, STATIC, AllTypes, rep_fixed32, rep_bool, 0),
-    PB_FIELD( 29, SFIXED32, REPEATED, STATIC, AllTypes, rep_sfixed32, rep_fixed32, 0),
-    PB_FIELD( 30, FLOAT   , REPEATED, STATIC, AllTypes, rep_float, rep_sfixed32, 0),
-    PB_FIELD( 31, FIXED64 , REPEATED, STATIC, AllTypes, rep_fixed64, rep_float, 0),
-    PB_FIELD( 32, SFIXED64, REPEATED, STATIC, AllTypes, rep_sfixed64, rep_fixed64, 0),
-    PB_FIELD( 33, DOUBLE  , REPEATED, STATIC, AllTypes, rep_double, rep_sfixed64, 0),
-    PB_FIELD( 34, STRING  , REPEATED, STATIC, AllTypes, rep_string, rep_double, 0),
-    PB_FIELD( 35, BYTES   , REPEATED, STATIC, AllTypes, rep_bytes, rep_string, 0),
-    PB_FIELD( 36, MESSAGE , REPEATED, STATIC, AllTypes, rep_submsg, rep_bytes, &SubMessage_fields),
-    PB_FIELD( 37, ENUM    , REPEATED, STATIC, AllTypes, rep_enum, rep_submsg, 0),
-    PB_FIELD( 41, INT32   , OPTIONAL, STATIC, AllTypes, opt_int32, rep_enum, &AllTypes_opt_int32_default),
-    PB_FIELD( 42, INT64   , OPTIONAL, STATIC, AllTypes, opt_int64, opt_int32, &AllTypes_opt_int64_default),
-    PB_FIELD( 43, UINT32  , OPTIONAL, STATIC, AllTypes, opt_uint32, opt_int64, &AllTypes_opt_uint32_default),
-    PB_FIELD( 44, UINT64  , OPTIONAL, STATIC, AllTypes, opt_uint64, opt_uint32, &AllTypes_opt_uint64_default),
-    PB_FIELD( 45, SINT32  , OPTIONAL, STATIC, AllTypes, opt_sint32, opt_uint64, &AllTypes_opt_sint32_default),
-    PB_FIELD( 46, SINT64  , OPTIONAL, STATIC, AllTypes, opt_sint64, opt_sint32, &AllTypes_opt_sint64_default),
-    PB_FIELD( 47, BOOL    , OPTIONAL, STATIC, AllTypes, opt_bool, opt_sint64, &AllTypes_opt_bool_default),
-    PB_FIELD( 48, FIXED32 , OPTIONAL, STATIC, AllTypes, opt_fixed32, opt_bool, &AllTypes_opt_fixed32_default),
-    PB_FIELD( 49, SFIXED32, OPTIONAL, STATIC, AllTypes, opt_sfixed32, opt_fixed32, &AllTypes_opt_sfixed32_default),
-    PB_FIELD( 50, FLOAT   , OPTIONAL, STATIC, AllTypes, opt_float, opt_sfixed32, &AllTypes_opt_float_default),
-    PB_FIELD( 51, FIXED64 , OPTIONAL, STATIC, AllTypes, opt_fixed64, opt_float, &AllTypes_opt_fixed64_default),
-    PB_FIELD( 52, SFIXED64, OPTIONAL, STATIC, AllTypes, opt_sfixed64, opt_fixed64, &AllTypes_opt_sfixed64_default),
-    PB_FIELD( 53, DOUBLE  , OPTIONAL, STATIC, AllTypes, opt_double, opt_sfixed64, &AllTypes_opt_double_default),
-    PB_FIELD( 54, STRING  , OPTIONAL, STATIC, AllTypes, opt_string, opt_double, &AllTypes_opt_string_default),
-    PB_FIELD( 55, BYTES   , OPTIONAL, STATIC, AllTypes, opt_bytes, opt_string, &AllTypes_opt_bytes_default),
-    PB_FIELD( 56, MESSAGE , OPTIONAL, STATIC, AllTypes, opt_submsg, opt_bytes, &SubMessage_fields),
-    PB_FIELD( 57, ENUM    , OPTIONAL, STATIC, AllTypes, opt_enum, opt_submsg, &AllTypes_opt_enum_default),
-    PB_FIELD( 99, INT32   , REQUIRED, STATIC, AllTypes, end, opt_enum, 0),
     PB_LAST_FIELD
 };
 
+const pb_field_t Limits_fields[11] = {
+    PB_FIELD(  1, INT32   , REQUIRED, STATIC  , FIRST, Limits, int32_min, int32_min, &Limits_int32_min_default),
+    PB_FIELD(  2, INT32   , REQUIRED, STATIC  , OTHER, Limits, int32_max, int32_min, &Limits_int32_max_default),
+    PB_FIELD(  3, UINT32  , REQUIRED, STATIC  , OTHER, Limits, uint32_min, int32_max, &Limits_uint32_min_default),
+    PB_FIELD(  4, UINT32  , REQUIRED, STATIC  , OTHER, Limits, uint32_max, uint32_min, &Limits_uint32_max_default),
+    PB_FIELD(  5, INT64   , REQUIRED, STATIC  , OTHER, Limits, int64_min, uint32_max, &Limits_int64_min_default),
+    PB_FIELD(  6, INT64   , REQUIRED, STATIC  , OTHER, Limits, int64_max, int64_min, &Limits_int64_max_default),
+    PB_FIELD(  7, UINT64  , REQUIRED, STATIC  , OTHER, Limits, uint64_min, int64_max, &Limits_uint64_min_default),
+    PB_FIELD(  8, UINT64  , REQUIRED, STATIC  , OTHER, Limits, uint64_max, uint64_min, &Limits_uint64_max_default),
+    PB_FIELD(  9, ENUM    , REQUIRED, STATIC  , OTHER, Limits, enum_min, uint64_max, &Limits_enum_min_default),
+    PB_FIELD( 10, ENUM    , REQUIRED, STATIC  , OTHER, Limits, enum_max, enum_min, &Limits_enum_max_default),
+    PB_LAST_FIELD
+};
+
+const pb_field_t AllTypes_fields[54] = {
+    PB_FIELD(  1, INT32   , REQUIRED, STATIC  , FIRST, AllTypes, req_int32, req_int32, 0),
+    PB_FIELD(  2, INT64   , REQUIRED, STATIC  , OTHER, AllTypes, req_int64, req_int32, 0),
+    PB_FIELD(  3, UINT32  , REQUIRED, STATIC  , OTHER, AllTypes, req_uint32, req_int64, 0),
+    PB_FIELD(  4, UINT64  , REQUIRED, STATIC  , OTHER, AllTypes, req_uint64, req_uint32, 0),
+    PB_FIELD(  5, SINT32  , REQUIRED, STATIC  , OTHER, AllTypes, req_sint32, req_uint64, 0),
+    PB_FIELD(  6, SINT64  , REQUIRED, STATIC  , OTHER, AllTypes, req_sint64, req_sint32, 0),
+    PB_FIELD(  7, BOOL    , REQUIRED, STATIC  , OTHER, AllTypes, req_bool, req_sint64, 0),
+    PB_FIELD(  8, FIXED32 , REQUIRED, STATIC  , OTHER, AllTypes, req_fixed32, req_bool, 0),
+    PB_FIELD(  9, SFIXED32, REQUIRED, STATIC  , OTHER, AllTypes, req_sfixed32, req_fixed32, 0),
+    PB_FIELD( 10, FLOAT   , REQUIRED, STATIC  , OTHER, AllTypes, req_float, req_sfixed32, 0),
+    PB_FIELD( 11, FIXED64 , REQUIRED, STATIC  , OTHER, AllTypes, req_fixed64, req_float, 0),
+    PB_FIELD( 12, SFIXED64, REQUIRED, STATIC  , OTHER, AllTypes, req_sfixed64, req_fixed64, 0),
+    PB_FIELD( 13, DOUBLE  , REQUIRED, STATIC  , OTHER, AllTypes, req_double, req_sfixed64, 0),
+    PB_FIELD( 14, STRING  , REQUIRED, STATIC  , OTHER, AllTypes, req_string, req_double, 0),
+    PB_FIELD( 15, BYTES   , REQUIRED, STATIC  , OTHER, AllTypes, req_bytes, req_string, 0),
+    PB_FIELD( 16, MESSAGE , REQUIRED, STATIC  , OTHER, AllTypes, req_submsg, req_bytes, &SubMessage_fields),
+    PB_FIELD( 17, ENUM    , REQUIRED, STATIC  , OTHER, AllTypes, req_enum, req_submsg, 0),
+    PB_FIELD( 21, INT32   , REPEATED, STATIC  , OTHER, AllTypes, rep_int32, req_enum, 0),
+    PB_FIELD( 22, INT64   , REPEATED, STATIC  , OTHER, AllTypes, rep_int64, rep_int32, 0),
+    PB_FIELD( 23, UINT32  , REPEATED, STATIC  , OTHER, AllTypes, rep_uint32, rep_int64, 0),
+    PB_FIELD( 24, UINT64  , REPEATED, STATIC  , OTHER, AllTypes, rep_uint64, rep_uint32, 0),
+    PB_FIELD( 25, SINT32  , REPEATED, STATIC  , OTHER, AllTypes, rep_sint32, rep_uint64, 0),
+    PB_FIELD( 26, SINT64  , REPEATED, STATIC  , OTHER, AllTypes, rep_sint64, rep_sint32, 0),
+    PB_FIELD( 27, BOOL    , REPEATED, STATIC  , OTHER, AllTypes, rep_bool, rep_sint64, 0),
+    PB_FIELD( 28, FIXED32 , REPEATED, STATIC  , OTHER, AllTypes, rep_fixed32, rep_bool, 0),
+    PB_FIELD( 29, SFIXED32, REPEATED, STATIC  , OTHER, AllTypes, rep_sfixed32, rep_fixed32, 0),
+    PB_FIELD( 30, FLOAT   , REPEATED, STATIC  , OTHER, AllTypes, rep_float, rep_sfixed32, 0),
+    PB_FIELD( 31, FIXED64 , REPEATED, STATIC  , OTHER, AllTypes, rep_fixed64, rep_float, 0),
+    PB_FIELD( 32, SFIXED64, REPEATED, STATIC  , OTHER, AllTypes, rep_sfixed64, rep_fixed64, 0),
+    PB_FIELD( 33, DOUBLE  , REPEATED, STATIC  , OTHER, AllTypes, rep_double, rep_sfixed64, 0),
+    PB_FIELD( 34, STRING  , REPEATED, STATIC  , OTHER, AllTypes, rep_string, rep_double, 0),
+    PB_FIELD( 35, BYTES   , REPEATED, STATIC  , OTHER, AllTypes, rep_bytes, rep_string, 0),
+    PB_FIELD( 36, MESSAGE , REPEATED, STATIC  , OTHER, AllTypes, rep_submsg, rep_bytes, &SubMessage_fields),
+    PB_FIELD( 37, ENUM    , REPEATED, STATIC  , OTHER, AllTypes, rep_enum, rep_submsg, 0),
+    PB_FIELD( 41, INT32   , OPTIONAL, STATIC  , OTHER, AllTypes, opt_int32, rep_enum, &AllTypes_opt_int32_default),
+    PB_FIELD( 42, INT64   , OPTIONAL, STATIC  , OTHER, AllTypes, opt_int64, opt_int32, &AllTypes_opt_int64_default),
+    PB_FIELD( 43, UINT32  , OPTIONAL, STATIC  , OTHER, AllTypes, opt_uint32, opt_int64, &AllTypes_opt_uint32_default),
+    PB_FIELD( 44, UINT64  , OPTIONAL, STATIC  , OTHER, AllTypes, opt_uint64, opt_uint32, &AllTypes_opt_uint64_default),
+    PB_FIELD( 45, SINT32  , OPTIONAL, STATIC  , OTHER, AllTypes, opt_sint32, opt_uint64, &AllTypes_opt_sint32_default),
+    PB_FIELD( 46, SINT64  , OPTIONAL, STATIC  , OTHER, AllTypes, opt_sint64, opt_sint32, &AllTypes_opt_sint64_default),
+    PB_FIELD( 47, BOOL    , OPTIONAL, STATIC  , OTHER, AllTypes, opt_bool, opt_sint64, &AllTypes_opt_bool_default),
+    PB_FIELD( 48, FIXED32 , OPTIONAL, STATIC  , OTHER, AllTypes, opt_fixed32, opt_bool, &AllTypes_opt_fixed32_default),
+    PB_FIELD( 49, SFIXED32, OPTIONAL, STATIC  , OTHER, AllTypes, opt_sfixed32, opt_fixed32, &AllTypes_opt_sfixed32_default),
+    PB_FIELD( 50, FLOAT   , OPTIONAL, STATIC  , OTHER, AllTypes, opt_float, opt_sfixed32, &AllTypes_opt_float_default),
+    PB_FIELD( 51, FIXED64 , OPTIONAL, STATIC  , OTHER, AllTypes, opt_fixed64, opt_float, &AllTypes_opt_fixed64_default),
+    PB_FIELD( 52, SFIXED64, OPTIONAL, STATIC  , OTHER, AllTypes, opt_sfixed64, opt_fixed64, &AllTypes_opt_sfixed64_default),
+    PB_FIELD( 53, DOUBLE  , OPTIONAL, STATIC  , OTHER, AllTypes, opt_double, opt_sfixed64, &AllTypes_opt_double_default),
+    PB_FIELD( 54, STRING  , OPTIONAL, STATIC  , OTHER, AllTypes, opt_string, opt_double, &AllTypes_opt_string_default),
+    PB_FIELD( 55, BYTES   , OPTIONAL, STATIC  , OTHER, AllTypes, opt_bytes, opt_string, &AllTypes_opt_bytes_default),
+    PB_FIELD( 56, MESSAGE , OPTIONAL, STATIC  , OTHER, AllTypes, opt_submsg, opt_bytes, &SubMessage_fields),
+    PB_FIELD( 57, ENUM    , OPTIONAL, STATIC  , OTHER, AllTypes, opt_enum, opt_submsg, &AllTypes_opt_enum_default),
+    PB_FIELD( 99, INT32   , REQUIRED, STATIC  , OTHER, AllTypes, end, opt_enum, 0),
+    PB_FIELD(200, EXTENSION, OPTIONAL, CALLBACK, OTHER, AllTypes, extensions, end, 0),
+    PB_LAST_FIELD
+};
+
+
+/* Check that field information fits in pb_field_t */
+#if !defined(PB_FIELD_32BIT)
+/* If you get an error here, it means that you need to define PB_FIELD_32BIT
+ * compile-time option. You can do that in pb.h or on compiler command line.
+ * 
+ * The reason you need to do this is that some of your messages contain tag
+ * numbers or field sizes that are larger than what can fit in 8 or 16 bit
+ * field descriptors.
+ */
+PB_STATIC_ASSERT((pb_membersize(AllTypes, req_submsg) < 65536 && pb_membersize(AllTypes, rep_submsg[0]) < 65536 && pb_membersize(AllTypes, opt_submsg) < 65536), YOU_MUST_DEFINE_PB_FIELD_32BIT_FOR_MESSAGES_SubMessage_EmptyMessage_Limits_AllTypes)
+#endif
+
+#if !defined(PB_FIELD_16BIT) && !defined(PB_FIELD_32BIT)
+/* If you get an error here, it means that you need to define PB_FIELD_16BIT
+ * compile-time option. You can do that in pb.h or on compiler command line.
+ * 
+ * The reason you need to do this is that some of your messages contain tag
+ * numbers or field sizes that are larger than what can fit in the default
+ * 8 bit descriptors.
+ */
+PB_STATIC_ASSERT((pb_membersize(AllTypes, req_submsg) < 256 && pb_membersize(AllTypes, rep_submsg[0]) < 256 && pb_membersize(AllTypes, opt_submsg) < 256), YOU_MUST_DEFINE_PB_FIELD_16BIT_FOR_MESSAGES_SubMessage_EmptyMessage_Limits_AllTypes)
+#endif
+
+
+/* On some platforms (such as AVR), double is really float.
+ * These are not directly supported by nanopb, but see example_avr_double.
+ * To get rid of this error, remove any double fields from your .proto.
+ */
+PB_STATIC_ASSERT(sizeof(double) == 8, DOUBLE_MUST_BE_8_BYTES)
+

tests/backwards_compatibility/alltypes_legacy.h

@@ -1,18 +1,24 @@
 /* Automatically generated nanopb header */
-/* This is a file generated using nanopb-0.2.0-dev.
+/* Generated by nanopb-0.3.0-dev at Tue Aug 19 17:53:24 2014. */
- * It is used as a part of test suite in order to detect any
- * incompatible changes made to the generator in future versions.
- */
 
-#ifndef _PB_ALLTYPES_PB_H_
+#ifndef PB_ALLTYPES_LEGACY_H_INCLUDED
-#define _PB_ALLTYPES_PB_H_
+#define PB_ALLTYPES_LEGACY_H_INCLUDED
 #include <pb.h>
 
+#if PB_PROTO_HEADER_VERSION != 30
+#error Regenerate this file with the current version of nanopb generator.
+#endif
+
 #ifdef __cplusplus
 extern "C" {
 #endif
 
 /* Enum definitions */
+typedef enum _HugeEnum {
+    HugeEnum_Negative = -2147483647,
+    HugeEnum_Positive = 2147483647
+} HugeEnum;
+
 typedef enum _MyEnum {
     MyEnum_Zero = 0,
     MyEnum_First = 1,
@@ -21,6 +27,23 @@ typedef enum _MyEnum {
 } MyEnum;
 
 /* Struct definitions */
+typedef struct _EmptyMessage {
+    uint8_t dummy_field;
+} EmptyMessage;
+
+typedef struct _Limits {
+    int32_t int32_min;
+    int32_t int32_max;
+    uint32_t uint32_min;
+    uint32_t uint32_max;
+    int64_t int64_min;
+    int64_t int64_max;
+    uint64_t uint64_min;
+    uint64_t uint64_max;
+    HugeEnum enum_min;
+    HugeEnum enum_max;
+} Limits;
+
 typedef struct _SubMessage {
     char substuff1[16];
     int32_t substuff2;
@@ -28,20 +51,11 @@ typedef struct _SubMessage {
     uint32_t substuff3;
 } SubMessage;
 
-typedef struct {
+typedef PB_BYTES_ARRAY_T(16) AllTypes_req_bytes_t;
-    size_t size;
-    uint8_t bytes[16];
-} AllTypes_req_bytes_t;
 
-typedef struct {
+typedef PB_BYTES_ARRAY_T(16) AllTypes_rep_bytes_t;
-    size_t size;
-    uint8_t bytes[16];
-} AllTypes_rep_bytes_t;
 
-typedef struct {
+typedef PB_BYTES_ARRAY_T(16) AllTypes_opt_bytes_t;
-    size_t size;
-    uint8_t bytes[16];
-} AllTypes_opt_bytes_t;
 
 typedef struct _AllTypes {
     int32_t req_int32;
@@ -61,39 +75,39 @@ typedef struct _AllTypes {
     AllTypes_req_bytes_t req_bytes;
     SubMessage req_submsg;
     MyEnum req_enum;
-    size_t rep_int32_count;
+    pb_size_t rep_int32_count;
     int32_t rep_int32[5];
-    size_t rep_int64_count;
+    pb_size_t rep_int64_count;
     int64_t rep_int64[5];
-    size_t rep_uint32_count;
+    pb_size_t rep_uint32_count;
     uint32_t rep_uint32[5];
-    size_t rep_uint64_count;
+    pb_size_t rep_uint64_count;
     uint64_t rep_uint64[5];
-    size_t rep_sint32_count;
+    pb_size_t rep_sint32_count;
     int32_t rep_sint32[5];
-    size_t rep_sint64_count;
+    pb_size_t rep_sint64_count;
     int64_t rep_sint64[5];
-    size_t rep_bool_count;
+    pb_size_t rep_bool_count;
     bool rep_bool[5];
-    size_t rep_fixed32_count;
+    pb_size_t rep_fixed32_count;
     uint32_t rep_fixed32[5];
-    size_t rep_sfixed32_count;
+    pb_size_t rep_sfixed32_count;
     int32_t rep_sfixed32[5];
-    size_t rep_float_count;
+    pb_size_t rep_float_count;
     float rep_float[5];
-    size_t rep_fixed64_count;
+    pb_size_t rep_fixed64_count;
     uint64_t rep_fixed64[5];
-    size_t rep_sfixed64_count;
+    pb_size_t rep_sfixed64_count;
     int64_t rep_sfixed64[5];
-    size_t rep_double_count;
+    pb_size_t rep_double_count;
     double rep_double[5];
-    size_t rep_string_count;
+    pb_size_t rep_string_count;
     char rep_string[5][16];
-    size_t rep_bytes_count;
+    pb_size_t rep_bytes_count;
     AllTypes_rep_bytes_t rep_bytes[5];
-    size_t rep_submsg_count;
+    pb_size_t rep_submsg_count;
     SubMessage rep_submsg[5];
-    size_t rep_enum_count;
+    pb_size_t rep_enum_count;
     MyEnum rep_enum[5];
     bool has_opt_int32;
     int32_t opt_int32;
@@ -130,12 +144,23 @@ typedef struct _AllTypes {
     bool has_opt_enum;
     MyEnum opt_enum;
     int32_t end;
+    pb_extension_t *extensions;
 } AllTypes;
 
 /* Default values for struct fields */
 extern const char SubMessage_substuff1_default[16];
 extern const int32_t SubMessage_substuff2_default;
 extern const uint32_t SubMessage_substuff3_default;
+extern const int32_t Limits_int32_min_default;
+extern const int32_t Limits_int32_max_default;
+extern const uint32_t Limits_uint32_min_default;
+extern const uint32_t Limits_uint32_max_default;
+extern const int64_t Limits_int64_min_default;
+extern const int64_t Limits_int64_max_default;
+extern const uint64_t Limits_uint64_min_default;
+extern const uint64_t Limits_uint64_max_default;
+extern const HugeEnum Limits_enum_min_default;
+extern const HugeEnum Limits_enum_max_default;
 extern const int32_t AllTypes_opt_int32_default;
 extern const int64_t AllTypes_opt_int64_default;
 extern const uint32_t AllTypes_opt_uint32_default;
@@ -153,23 +178,94 @@ extern const char AllTypes_opt_string_default[16];
 extern const AllTypes_opt_bytes_t AllTypes_opt_bytes_default;
 extern const MyEnum AllTypes_opt_enum_default;
 
+/* Initializer values for message structs */
+#define SubMessage_init_default                  {"1", 2, false, 3u}
+#define EmptyMessage_init_default                {0}
+#define Limits_init_default                      {2147483647, -2147483647, 4294967295u, 0u, 9223372036854775807ll, -9223372036854775807ll, 18446744073709551615ull, 0ull, HugeEnum_Positive, HugeEnum_Negative}
+#define AllTypes_init_default                    {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, "", {0, {0}}, SubMessage_init_default, (MyEnum)0, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {"", "", "", "", ""}, 0, {{0, {0}}, {0, {0}}, {0, {0}}, {0, {0}}, {0, {0}}}, 0, {SubMessage_init_default, SubMessage_init_default, SubMessage_init_default, SubMessage_init_default, SubMessage_init_default}, 0, {(MyEnum)0, (MyEnum)0, (MyEnum)0, (MyEnum)0, (MyEnum)0}, false, 4041, false, 4042ll, false, 4043u, false, 4044ull, false, 4045, false, 4046, false, false, false, 4048u, false, 4049, false, 4050, false, 4051ull, false, 4052ll, false, 4053, false, "4054", false, {4, {0x34,0x30,0x35,0x35}}, false, SubMessage_init_default, false, MyEnum_Second, 0, NULL}
+#define SubMessage_init_zero                     {"", 0, false, 0}
+#define EmptyMessage_init_zero                   {0}
+#define Limits_init_zero                         {0, 0, 0, 0, 0, 0, 0, 0, (HugeEnum)0, (HugeEnum)0}
+#define AllTypes_init_zero                       {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, "", {0, {0}}, SubMessage_init_zero, (MyEnum)0, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {"", "", "", "", ""}, 0, {{0, {0}}, {0, {0}}, {0, {0}}, {0, {0}}, {0, {0}}}, 0, {SubMessage_init_zero, SubMessage_init_zero, SubMessage_init_zero, SubMessage_init_zero, SubMessage_init_zero}, 0, {(MyEnum)0, (MyEnum)0, (MyEnum)0, (MyEnum)0, (MyEnum)0}, false, 0, false, 0, false, 0, false, 0, false, 0, false, 0, false, 0, false, 0, false, 0, false, 0, false, 0, false, 0, false, 0, false, "", false, {0, {0}}, false, SubMessage_init_zero, false, (MyEnum)0, 0, NULL}
+
+/* Field tags (for use in manual encoding/decoding) */
+#define Limits_int32_min_tag                     1
+#define Limits_int32_max_tag                     2
+#define Limits_uint32_min_tag                    3
+#define Limits_uint32_max_tag                    4
+#define Limits_int64_min_tag                     5
+#define Limits_int64_max_tag                     6
+#define Limits_uint64_min_tag                    7
+#define Limits_uint64_max_tag                    8
+#define Limits_enum_min_tag                      9
+#define Limits_enum_max_tag                      10
+#define SubMessage_substuff1_tag                 1
+#define SubMessage_substuff2_tag                 2
+#define SubMessage_substuff3_tag                 3
+#define AllTypes_req_int32_tag                   1
+#define AllTypes_req_int64_tag                   2
+#define AllTypes_req_uint32_tag                  3
+#define AllTypes_req_uint64_tag                  4
+#define AllTypes_req_sint32_tag                  5
+#define AllTypes_req_sint64_tag                  6
+#define AllTypes_req_bool_tag                    7
+#define AllTypes_req_fixed32_tag                 8
+#define AllTypes_req_sfixed32_tag                9
+#define AllTypes_req_float_tag                   10
+#define AllTypes_req_fixed64_tag                 11
+#define AllTypes_req_sfixed64_tag                12
+#define AllTypes_req_double_tag                  13
+#define AllTypes_req_string_tag                  14
+#define AllTypes_req_bytes_tag                   15
+#define AllTypes_req_submsg_tag                  16
+#define AllTypes_req_enum_tag                    17
+#define AllTypes_rep_int32_tag                   21
+#define AllTypes_rep_int64_tag                   22
+#define AllTypes_rep_uint32_tag                  23
+#define AllTypes_rep_uint64_tag                  24
+#define AllTypes_rep_sint32_tag                  25
+#define AllTypes_rep_sint64_tag                  26
+#define AllTypes_rep_bool_tag                    27
+#define AllTypes_rep_fixed32_tag                 28
+#define AllTypes_rep_sfixed32_tag                29
+#define AllTypes_rep_float_tag                   30
+#define AllTypes_rep_fixed64_tag                 31
+#define AllTypes_rep_sfixed64_tag                32
+#define AllTypes_rep_double_tag                  33
+#define AllTypes_rep_string_tag                  34
+#define AllTypes_rep_bytes_tag                   35
+#define AllTypes_rep_submsg_tag                  36
+#define AllTypes_rep_enum_tag                    37
+#define AllTypes_opt_int32_tag                   41
+#define AllTypes_opt_int64_tag                   42
+#define AllTypes_opt_uint32_tag                  43
+#define AllTypes_opt_uint64_tag                  44
+#define AllTypes_opt_sint32_tag                  45
+#define AllTypes_opt_sint64_tag                  46
+#define AllTypes_opt_bool_tag                    47
+#define AllTypes_opt_fixed32_tag                 48
+#define AllTypes_opt_sfixed32_tag                49
+#define AllTypes_opt_float_tag                   50
+#define AllTypes_opt_fixed64_tag                 51
+#define AllTypes_opt_sfixed64_tag                52
+#define AllTypes_opt_double_tag                  53
+#define AllTypes_opt_string_tag                  54
+#define AllTypes_opt_bytes_tag                   55
+#define AllTypes_opt_submsg_tag                  56
+#define AllTypes_opt_enum_tag                    57
+#define AllTypes_end_tag                         99
+
 /* Struct field encoding specification for nanopb */
 extern const pb_field_t SubMessage_fields[4];
-extern const pb_field_t AllTypes_fields[53];
+extern const pb_field_t EmptyMessage_fields[1];
+extern const pb_field_t Limits_fields[11];
+extern const pb_field_t AllTypes_fields[54];
 
-/* Check that field information fits in pb_field_t */
+/* Maximum encoded size of messages (where known) */
-#if !defined(PB_FIELD_16BIT) && !defined(PB_FIELD_32BIT)
+#define SubMessage_size                          34
-STATIC_ASSERT((pb_membersize(AllTypes, req_submsg) < 256 && pb_membersize(AllTypes, rep_submsg[0]) < 256 && pb_membersize(AllTypes, opt_submsg) < 256), YOU_MUST_DEFINE_PB_FIELD_16BIT_FOR_MESSAGES_SubMessage_AllTypes)
+#define EmptyMessage_size                        0
-#endif
+#define Limits_size                              90
-
+#define AllTypes_size                            1362
-#if !defined(PB_FIELD_32BIT)
-STATIC_ASSERT((pb_membersize(AllTypes, req_submsg) < 65536 && pb_membersize(AllTypes, rep_submsg[0]) < 65536 && pb_membersize(AllTypes, opt_submsg) < 65536), YOU_MUST_DEFINE_PB_FIELD_32BIT_FOR_MESSAGES_SubMessage_AllTypes)
-#endif
-
-/* On some platforms (such as AVR), double is really float.
- * These are not directly supported by nanopb, but see example_avr_double.
- */
-STATIC_ASSERT(sizeof(double) == 8, DOUBLE_MUST_BE_8_BYTES)
 
 #ifdef __cplusplus
 } /* extern "C" */

tests/backwards_compatibility/alltypes_legacy.options

@@ -0,0 +1,3 @@
+* max_size:16
+* max_count:5
+

tests/backwards_compatibility/alltypes_legacy.proto

@@ -0,0 +1,108 @@
+message SubMessage {
+    required string substuff1 = 1 [default = "1"];
+    required int32 substuff2 = 2 [default = 2];
+    optional fixed32 substuff3 = 3 [default = 3];
+}
+
+message EmptyMessage {
+
+}
+
+enum HugeEnum {
+    Negative = -2147483647; /* protoc doesn't accept -2147483648 here */
+    Positive =  2147483647;
+}
+
+message Limits {
+    required int32      int32_min  =  1 [default = 2147483647];
+    required int32      int32_max  =  2 [default = -2147483647];
+    required uint32     uint32_min =  3 [default = 4294967295];
+    required uint32     uint32_max =  4 [default = 0];
+    required int64      int64_min  =  5 [default = 9223372036854775807];
+    required int64      int64_max  =  6 [default = -9223372036854775807];
+    required uint64     uint64_min =  7 [default = 18446744073709551615];
+    required uint64     uint64_max =  8 [default = 0];
+    required HugeEnum   enum_min   =  9 [default = Positive];
+    required HugeEnum   enum_max   = 10 [default = Negative];
+}
+
+enum MyEnum {
+    Zero = 0;
+    First = 1;
+    Second = 2;
+    Truth = 42;
+}
+
+message AllTypes {
+    required int32      req_int32   = 1;
+    required int64      req_int64   = 2;
+    required uint32     req_uint32  = 3;
+    required uint64     req_uint64  = 4;
+    required sint32     req_sint32  = 5;
+    required sint64     req_sint64  = 6;
+    required bool       req_bool    = 7;
+    
+    required fixed32    req_fixed32 = 8;
+    required sfixed32   req_sfixed32= 9;
+    required float      req_float   = 10;
+    
+    required fixed64    req_fixed64 = 11;
+    required sfixed64   req_sfixed64= 12;
+    required double     req_double  = 13;
+    
+    required string     req_string  = 14;
+    required bytes      req_bytes   = 15;
+    required SubMessage req_submsg  = 16;
+    required MyEnum     req_enum    = 17;
+    
+    
+    repeated int32      rep_int32   = 21 [packed = true];
+    repeated int64      rep_int64   = 22 [packed = true];
+    repeated uint32     rep_uint32  = 23 [packed = true];
+    repeated uint64     rep_uint64  = 24 [packed = true];
+    repeated sint32     rep_sint32  = 25 [packed = true];
+    repeated sint64     rep_sint64  = 26 [packed = true];
+    repeated bool       rep_bool    = 27 [packed = true];
+    
+    repeated fixed32    rep_fixed32 = 28 [packed = true];
+    repeated sfixed32   rep_sfixed32= 29 [packed = true];
+    repeated float      rep_float   = 30 [packed = true];
+    
+    repeated fixed64    rep_fixed64 = 31 [packed = true];
+    repeated sfixed64   rep_sfixed64= 32 [packed = true];
+    repeated double     rep_double  = 33 [packed = true];
+    
+    repeated string     rep_string  = 34;
+    repeated bytes      rep_bytes   = 35;
+    repeated SubMessage rep_submsg  = 36;
+    repeated MyEnum     rep_enum    = 37 [packed = true];
+    
+    optional int32      opt_int32   = 41 [default = 4041];
+    optional int64      opt_int64   = 42 [default = 4042];
+    optional uint32     opt_uint32  = 43 [default = 4043];
+    optional uint64     opt_uint64  = 44 [default = 4044];
+    optional sint32     opt_sint32  = 45 [default = 4045];
+    optional sint64     opt_sint64  = 46 [default = 4046];
+    optional bool       opt_bool    = 47 [default = false];
+    
+    optional fixed32    opt_fixed32 = 48 [default = 4048];
+    optional sfixed32   opt_sfixed32= 49 [default = 4049];
+    optional float      opt_float   = 50 [default = 4050];
+    
+    optional fixed64    opt_fixed64 = 51 [default = 4051];
+    optional sfixed64   opt_sfixed64= 52 [default = 4052];
+    optional double     opt_double  = 53 [default = 4053];
+    
+    optional string     opt_string  = 54 [default = "4054"];
+    optional bytes      opt_bytes   = 55 [default = "4055"];
+    optional SubMessage opt_submsg  = 56;
+    optional MyEnum     opt_enum    = 57 [default = Second];
+
+    // Just to make sure that the size of the fields has been calculated
+    // properly, i.e. otherwise a bug in last field might not be detected.
+    required int32      end = 99;
+
+
+    extensions 200 to 255;
+}
+

tests/basic_buffer/SConscript

@@ -2,8 +2,8 @@
 
 Import("env")
 
-enc = env.Program(["encode_buffer.c", "$COMMON/person.pb.c", "$COMMON/pb_encode.o"])
+enc = env.Program(["encode_buffer.c", "$COMMON/person.pb.c", "$COMMON/pb_encode.o", "$COMMON/pb_common.o"])
-dec = env.Program(["decode_buffer.c", "$COMMON/person.pb.c", "$COMMON/pb_decode.o"])
+dec = env.Program(["decode_buffer.c", "$COMMON/person.pb.c", "$COMMON/pb_decode.o", "$COMMON/pb_common.o"])
 
 env.RunTest(enc)
 env.RunTest([dec, "encode_buffer.output"])

tests/basic_buffer/decode_buffer.c

@@ -16,7 +16,7 @@
 bool print_person(pb_istream_t *stream)
 {
     int i;
-    Person person;
+    Person person = Person_init_zero;
     
     if (!pb_decode(stream, Person_fields, &person))
         return false;

tests/basic_stream/SConscript

@@ -2,8 +2,8 @@
 
 Import("env")
 
-enc = env.Program(["encode_stream.c", "$COMMON/person.pb.c", "$COMMON/pb_encode.o"])
+enc = env.Program(["encode_stream.c", "$COMMON/person.pb.c", "$COMMON/pb_encode.o", "$COMMON/pb_common.o"])
-dec = env.Program(["decode_stream.c", "$COMMON/person.pb.c", "$COMMON/pb_decode.o"])
+dec = env.Program(["decode_stream.c", "$COMMON/person.pb.c", "$COMMON/pb_decode.o", "$COMMON/pb_common.o"])
 
 env.RunTest(enc)
 env.RunTest([dec, "encode_stream.output"])

tests/basic_stream/decode_stream.c

@@ -12,7 +12,7 @@
 bool print_person(pb_istream_t *stream)
 {
     int i;
-    Person person;
+    Person person = Person_init_zero;
     
     if (!pb_decode(stream, Person_fields, &person))
         return false;

tests/buffer_only/SConscript

@@ -4,8 +4,6 @@ Import("env")
 
 # Take copy of the files for custom build.
 c = Copy("$TARGET", "$SOURCE")
-env.Command("pb_encode.c", "#../pb_encode.c", c)
-env.Command("pb_decode.c", "#../pb_decode.c", c)
 env.Command("alltypes.pb.h", "$BUILD/alltypes/alltypes.pb.h", c)
 env.Command("alltypes.pb.c", "$BUILD/alltypes/alltypes.pb.c", c)
 env.Command("encode_alltypes.c", "$BUILD/alltypes/encode_alltypes.c", c)
@@ -15,9 +13,16 @@ env.Command("decode_alltypes.c", "$BUILD/alltypes/decode_alltypes.c", c)
 opts = env.Clone()
 opts.Append(CPPDEFINES = {'PB_BUFFER_ONLY': 1})
 
+# Build new version of core
+strict = opts.Clone()
+strict.Append(CFLAGS = strict['CORECFLAGS'])
+strict.Object("pb_decode_bufonly.o", "$NANOPB/pb_decode.c")
+strict.Object("pb_encode_bufonly.o", "$NANOPB/pb_encode.c")
+strict.Object("pb_common_bufonly.o", "$NANOPB/pb_common.c")
+
 # Now build and run the test normally.
-enc = opts.Program(["encode_alltypes.c", "alltypes.pb.c", "pb_encode.c"])
+enc = opts.Program(["encode_alltypes.c", "alltypes.pb.c", "pb_encode_bufonly.o", "pb_common_bufonly.o"])
-dec = opts.Program(["decode_alltypes.c", "alltypes.pb.c", "pb_decode.c"])
+dec = opts.Program(["decode_alltypes.c", "alltypes.pb.c", "pb_decode_bufonly.o", "pb_common_bufonly.o"])
 
 env.RunTest(enc)
 env.RunTest([dec, "encode_alltypes.output"])

tests/callbacks/SConscript

@@ -3,8 +3,8 @@
 Import("env")
 
 env.NanopbProto("callbacks")
-enc = env.Program(["encode_callbacks.c", "callbacks.pb.c", "$COMMON/pb_encode.o"])
+enc = env.Program(["encode_callbacks.c", "callbacks.pb.c", "$COMMON/pb_encode.o", "$COMMON/pb_common.o"])
-dec = env.Program(["decode_callbacks.c", "callbacks.pb.c", "$COMMON/pb_decode.o"])
+dec = env.Program(["decode_callbacks.c", "callbacks.pb.c", "$COMMON/pb_decode.o", "$COMMON/pb_common.o"])
 
 env.RunTest(enc)
 env.RunTest([dec, "encode_callbacks.output"])

tests/common/SConscript

@@ -14,4 +14,4 @@ strict = env.Clone()
 strict.Append(CFLAGS = strict['CORECFLAGS'])
 strict.Object("pb_decode.o", "$NANOPB/pb_decode.c")
 strict.Object("pb_encode.o", "$NANOPB/pb_encode.c")
-
+strict.Object("pb_common.o", "$NANOPB/pb_common.c")

tests/cxx_main_program/SConscript

@@ -11,14 +11,15 @@ env.Append(CPPDEFINES = ['__STDC_LIMIT_MACROS'])
 c = Copy("$TARGET", "$SOURCE")
 env.Command("pb_encode.cxx", "#../pb_encode.c", c)
 env.Command("pb_decode.cxx", "#../pb_decode.c", c)
+env.Command("pb_common.cxx", "#../pb_common.c", c)
 env.Command("alltypes.pb.h", "$BUILD/alltypes/alltypes.pb.h", c)
 env.Command("alltypes.pb.cxx", "$BUILD/alltypes/alltypes.pb.c", c)
 env.Command("encode_alltypes.cxx", "$BUILD/alltypes/encode_alltypes.c", c)
 env.Command("decode_alltypes.cxx", "$BUILD/alltypes/decode_alltypes.c", c)
 
 # Now build and run the test normally.
-enc = env.Program(["encode_alltypes.cxx", "alltypes.pb.cxx", "pb_encode.cxx"])
+enc = env.Program(["encode_alltypes.cxx", "alltypes.pb.cxx", "pb_encode.cxx", "pb_common.cxx"])
-dec = env.Program(["decode_alltypes.cxx", "alltypes.pb.cxx", "pb_decode.cxx"])
+dec = env.Program(["decode_alltypes.cxx", "alltypes.pb.cxx", "pb_decode.cxx", "pb_common.cxx"])
 
 env.RunTest(enc)
 env.RunTest([dec, "encode_alltypes.output"])

tests/decode_unittests/decode_unittests.c

@@ -1,4 +1,6 @@
 /* This includes the whole .c file to get access to static functions. */
+#define PB_ENABLE_MALLOC
+#include "pb_common.c"
 #include "pb_decode.c"
 
 #include <stdio.h>
@@ -168,7 +170,7 @@ int main()
     
     {
         pb_istream_t s;
-        struct { size_t size; uint8_t bytes[5]; } d;
+        struct { pb_size_t size; uint8_t bytes[5]; } d;
         pb_field_t f = {1, PB_LTYPE_BYTES, 0, 0, sizeof(d), 0, 0};
         
         COMMENT("Test pb_dec_bytes")
@@ -249,7 +251,7 @@ int main()
     {
         pb_istream_t s;
         CallbackArray dest;
-        struct { size_t size; uint8_t bytes[10]; } ref;
+        struct { pb_size_t size; uint8_t bytes[10]; } ref;
         dest.data.funcs.decode = &callback_check;
         dest.data.arg = &ref;
         
@@ -299,6 +301,28 @@ int main()
               dest.submsg.data_count == 5)
     }
     
+    {
+        pb_istream_t s = {0};
+        void *data = NULL;
+        
+        COMMENT("Testing allocate_field")
+        TEST(allocate_field(&s, &data, 10, 10) && data != NULL);
+        TEST(allocate_field(&s, &data, 10, 20) && data != NULL);
+        
+        {
+            void *oldvalue = data;
+            size_t very_big = (size_t)-1;
+            size_t somewhat_big = very_big / 2 + 1;
+            size_t not_so_big = (size_t)1 << (4 * sizeof(size_t));
+        
+            TEST(!allocate_field(&s, &data, very_big, 2) && data == oldvalue);
+            TEST(!allocate_field(&s, &data, somewhat_big, 2) && data == oldvalue);
+            TEST(!allocate_field(&s, &data, not_so_big, not_so_big) && data == oldvalue);
+        }
+        
+        pb_free(data);
+    }
+    
     if (status != 0)
         fprintf(stdout, "\n\nSome tests FAILED!\n");
     

tests/encode_unittests/encode_unittests.c

@@ -1,4 +1,5 @@
 /* This includes the whole .c file to get access to static functions. */
+#include "pb_common.c"
 #include "pb_encode.c"
 
 #include <stdio.h>
@@ -169,7 +170,7 @@ int main()
     {
         uint8_t buffer[30];
         pb_ostream_t s;
-        struct { size_t size; uint8_t bytes[5]; } value = {5, {'x', 'y', 'z', 'z', 'y'}};
+        struct { pb_size_t size; uint8_t bytes[5]; } value = {5, {'x', 'y', 'z', 'z', 'y'}};
     
         COMMENT("Test pb_enc_bytes")
         TEST(WRITES(pb_enc_bytes(&s, &BytesMessage_fields[0], &value), "\x05xyzzy"))
@@ -282,6 +283,15 @@ int main()
                     "\x09\x0A\x07\x0A\x05\x01\x02\x03\x04\x05"))
     }
 
+    {
+        IntegerContainer msg = {{5, {1,2,3,4,5}}};
+        size_t size;
+        
+        COMMENT("Test pb_get_encoded_size.")
+        TEST(pb_get_encoded_size(&size, IntegerContainer_fields, &msg) &&
+             size == 9);
+    }
+    
     {
         uint8_t buffer[10];
         pb_ostream_t s;

tests/extensions/SConscript

@@ -8,8 +8,8 @@ incpath.Append(PROTOCPATH = '$BUILD/alltypes')
 incpath.Append(CPPPATH = '$BUILD/alltypes')
 
 incpath.NanopbProto(["extensions", "extensions.options"])
-enc = incpath.Program(["encode_extensions.c", "extensions.pb.c", "$BUILD/alltypes/alltypes.pb$OBJSUFFIX", "$COMMON/pb_encode.o"])
+enc = incpath.Program(["encode_extensions.c", "extensions.pb.c", "$BUILD/alltypes/alltypes.pb$OBJSUFFIX", "$COMMON/pb_encode.o", "$COMMON/pb_common.o"])
-dec = incpath.Program(["decode_extensions.c", "extensions.pb.c", "$BUILD/alltypes/alltypes.pb$OBJSUFFIX", "$COMMON/pb_decode.o"])
+dec = incpath.Program(["decode_extensions.c", "extensions.pb.c", "$BUILD/alltypes/alltypes.pb$OBJSUFFIX", "$COMMON/pb_decode.o", "$COMMON/pb_common.o"])
 
 env.RunTest(enc)
 env.RunTest([dec, "encode_extensions.output"])

tests/extensions/decode_extensions.c

@@ -49,7 +49,9 @@ int main(int argc, char **argv)
     }
 
     /* Check that the extensions decoded properly */
+    TEST(ext1.found)
     TEST(extensionfield1 == 12345)
+    TEST(ext2.found)
     TEST(strcmp(extensionfield2.test1, "test") == 0)
     TEST(extensionfield2.test2 == 54321)
     

tests/extensions/extensions.proto

@@ -1,7 +1,7 @@
 import 'alltypes.proto';
 
 extend AllTypes {
-    optional int32 AllTypes_extensionfield1 = 255;
+    optional int32 AllTypes_extensionfield1 = 255 [default = 5];
 }
 
 message ExtensionMessage {

tests/field_size_16/SConscript

@@ -5,8 +5,6 @@ Import("env")
 
 # Take copy of the files for custom build.
 c = Copy("$TARGET", "$SOURCE")
-env.Command("pb_encode.c", "#../pb_encode.c", c)
-env.Command("pb_decode.c", "#../pb_decode.c", c)
 env.Command("encode_alltypes.c", "$BUILD/alltypes/encode_alltypes.c", c)
 env.Command("decode_alltypes.c", "$BUILD/alltypes/decode_alltypes.c", c)
 
@@ -16,9 +14,16 @@ env.NanopbProto(["alltypes", "alltypes.options"])
 opts = env.Clone()
 opts.Append(CPPDEFINES = {'PB_FIELD_16BIT': 1})
 
+# Build new version of core
+strict = opts.Clone()
+strict.Append(CFLAGS = strict['CORECFLAGS'])
+strict.Object("pb_decode_fields16.o", "$NANOPB/pb_decode.c")
+strict.Object("pb_encode_fields16.o", "$NANOPB/pb_encode.c")
+strict.Object("pb_common_fields16.o", "$NANOPB/pb_common.c")
+
 # Now build and run the test normally.
-enc = opts.Program(["encode_alltypes.c", "alltypes.pb.c", "pb_encode.c"])
+enc = opts.Program(["encode_alltypes.c", "alltypes.pb.c", "pb_encode_fields16.o", "pb_common_fields16.o"])
-dec = opts.Program(["decode_alltypes.c", "alltypes.pb.c", "pb_decode.c"])
+dec = opts.Program(["decode_alltypes.c", "alltypes.pb.c", "pb_decode_fields16.o", "pb_common_fields16.o"])
 
 env.RunTest(enc)
 env.RunTest([dec, "encode_alltypes.output"])

tests/field_size_16/alltypes.proto

@@ -107,5 +107,7 @@ message AllTypes {
     // Just to make sure that the size of the fields has been calculated
     // properly, i.e. otherwise a bug in last field might not be detected.
     required int32      end = 10099;
+
+    extensions 200 to 255;
 }
 

tests/field_size_32/SConscript

@@ -5,8 +5,6 @@ Import("env")
 
 # Take copy of the files for custom build.
 c = Copy("$TARGET", "$SOURCE")
-env.Command("pb_encode.c", "#../pb_encode.c", c)
-env.Command("pb_decode.c", "#../pb_decode.c", c)
 env.Command("encode_alltypes.c", "$BUILD/alltypes/encode_alltypes.c", c)
 env.Command("decode_alltypes.c", "$BUILD/alltypes/decode_alltypes.c", c)
 
@@ -16,9 +14,16 @@ env.NanopbProto(["alltypes", "alltypes.options"])
 opts = env.Clone()
 opts.Append(CPPDEFINES = {'PB_FIELD_32BIT': 1})
 
+# Build new version of core
+strict = opts.Clone()
+strict.Append(CFLAGS = strict['CORECFLAGS'])
+strict.Object("pb_decode_fields32.o", "$NANOPB/pb_decode.c")
+strict.Object("pb_encode_fields32.o", "$NANOPB/pb_encode.c")
+strict.Object("pb_common_fields32.o", "$NANOPB/pb_common.c")
+
 # Now build and run the test normally.
-enc = opts.Program(["encode_alltypes.c", "alltypes.pb.c", "pb_encode.c"])
+enc = opts.Program(["encode_alltypes.c", "alltypes.pb.c", "pb_encode_fields32.o", "pb_common_fields32.o"])
-dec = opts.Program(["decode_alltypes.c", "alltypes.pb.c", "pb_decode.c"])
+dec = opts.Program(["decode_alltypes.c", "alltypes.pb.c", "pb_decode_fields32.o", "pb_common_fields32.o"])
 
 env.RunTest(enc)
 env.RunTest([dec, "encode_alltypes.output"])

tests/field_size_32/alltypes.proto

@@ -107,5 +107,7 @@ message AllTypes {
     // Just to make sure that the size of the fields has been calculated
     // properly, i.e. otherwise a bug in last field might not be detected.
     required int32      end = 13432099;
+
+    extensions 200 to 255;
 }
 

tests/missing_fields/SConscript

@@ -3,6 +3,6 @@
 Import("env")
 
 env.NanopbProto("missing_fields")
-test = env.Program(["missing_fields.c", "missing_fields.pb.c", "$COMMON/pb_encode.o", "$COMMON/pb_decode.o"])
+test = env.Program(["missing_fields.c", "missing_fields.pb.c", "$COMMON/pb_encode.o", "$COMMON/pb_decode.o", "$COMMON/pb_common.o"])
 env.RunTest(test)
 

tests/missing_fields/missing_fields.c

@@ -8,6 +8,7 @@
 int main()
 {
     uint8_t buffer[512];
+    size_t size;
     
     /* Create a message with one missing field */
     {
@@ -19,12 +20,14 @@ int main()
             printf("Encode failed.\n");
             return 1;
         }
+
+        size = stream.bytes_written;
     }
 
     /* Test that it decodes properly if we don't require that field */
     {
         MissingField msg = {0};
-        pb_istream_t stream = pb_istream_from_buffer(buffer, sizeof(buffer));
+        pb_istream_t stream = pb_istream_from_buffer(buffer, size);
         
         if (!pb_decode(&stream, MissingField_fields, &msg))
         {
@@ -36,7 +39,7 @@ int main()
     /* Test that it does *not* decode properly if we require the field */
     {
         AllFields msg = {0};
-        pb_istream_t stream = pb_istream_from_buffer(buffer, sizeof(buffer));
+        pb_istream_t stream = pb_istream_from_buffer(buffer, size);
         
         if (pb_decode(&stream, AllFields_fields, &msg))
         {

tests/no_errmsg/SConscript

@@ -4,8 +4,6 @@ Import("env")
 
 # Take copy of the files for custom build.
 c = Copy("$TARGET", "$SOURCE")
-env.Command("pb_encode.c", "#../pb_encode.c", c)
-env.Command("pb_decode.c", "#../pb_decode.c", c)
 env.Command("alltypes.pb.h", "$BUILD/alltypes/alltypes.pb.h", c)
 env.Command("alltypes.pb.c", "$BUILD/alltypes/alltypes.pb.c", c)
 env.Command("encode_alltypes.c", "$BUILD/alltypes/encode_alltypes.c", c)
@@ -15,9 +13,16 @@ env.Command("decode_alltypes.c", "$BUILD/alltypes/decode_alltypes.c", c)
 opts = env.Clone()
 opts.Append(CPPDEFINES = {'PB_NO_ERRMSG': 1})
 
+# Build new version of core
+strict = opts.Clone()
+strict.Append(CFLAGS = strict['CORECFLAGS'])
+strict.Object("pb_decode_noerr.o", "$NANOPB/pb_decode.c")
+strict.Object("pb_encode_noerr.o", "$NANOPB/pb_encode.c")
+strict.Object("pb_common_noerr.o", "$NANOPB/pb_common.c")
+
 # Now build and run the test normally.
-enc = opts.Program(["encode_alltypes.c", "alltypes.pb.c", "pb_encode.c"])
+enc = opts.Program(["encode_alltypes.c", "alltypes.pb.c", "pb_encode_noerr.o", "pb_common_noerr.o"])
-dec = opts.Program(["decode_alltypes.c", "alltypes.pb.c", "pb_decode.c"])
+dec = opts.Program(["decode_alltypes.c", "alltypes.pb.c", "pb_decode_noerr.o", "pb_common_noerr.o"])
 
 env.RunTest(enc)
 env.RunTest([dec, "encode_alltypes.output"])

tests/options/options.expected

@@ -5,3 +5,6 @@ pb_callback_t int32_callback;
 \sEnumValue1 = 1
 Message5_EnumValue1
 } pb_packed my_packed_struct;
+! skipped_field
+! SkippedMessage
+

tests/options/options.proto

@@ -63,11 +63,15 @@ message my_packed_struct
 }
 
 // Message with ignored field
-// Note: doesn't really test if the field is missing in the output,
-// but atleast tests that the output compiles.
 message Message6
 {
     required int32 field1 = 1;
-    optional int32 field2 = 2 [(nanopb).type = FT_IGNORE];
+    optional int32 skipped_field = 2 [(nanopb).type = FT_IGNORE];
 }
 
+// Message that is skipped
+message SkippedMessage
+{
+    option (nanopb_msgopt).skip_message = true;
+    required int32 foo = 1;
+}

tests/package_name/SConscript

@@ -29,7 +29,7 @@ def modify_c(target, source, env):
 env.Command("encode_alltypes.c", "#alltypes/encode_alltypes.c", modify_c)
 
 # Encode and compare results to original alltypes testcase
-enc = env.Program(["encode_alltypes.c", "alltypes.pb.c", "$COMMON/pb_encode.o"])
+enc = env.Program(["encode_alltypes.c", "alltypes.pb.c", "$COMMON/pb_encode.o", "$COMMON/pb_common.o"])
 refdec = "$BUILD/alltypes/decode_alltypes$PROGSUFFIX"
 env.RunTest(enc)
 env.Compare(["encode_alltypes.output", "$BUILD/alltypes/encode_alltypes.output"])

tests/regression/issue_125/SConscript

@@ -0,0 +1,9 @@
+# Regression test for Issue 125: Wrong identifier name for extension fields
+
+Import("env")
+
+env.NanopbProto(["extensionbug", "extensionbug.options"])
+env.Object('extensionbug.pb.c')
+
+env.Match(['extensionbug.pb.h', 'extensionbug.expected'])
+

tests/regression/issue_125/extensionbug.expected

@@ -0,0 +1,3 @@
+pb_extension_type_t Message2_extras
+uint32_t field2
+

tests/regression/issue_125/extensionbug.options

@@ -0,0 +1,4 @@
+* type:FT_IGNORE
+
+Message2.extras type:FT_STATIC
+Message2.field2 type:FT_STATIC

tests/regression/issue_125/extensionbug.proto

@@ -0,0 +1,16 @@
+message Message1
+{
+	optional uint32	fieldA = 1;
+	extensions 30 to max;
+}
+
+message Message2
+{
+	extend Message1
+	{
+		optional Message2 extras = 30;
+	}
+	
+	optional uint32	field1 = 1;
+	optional uint32 field2 = 2;
+}

tests/site_scons/site_init.py

@@ -85,9 +85,20 @@ def add_nanopb_builders(env):
         data = open(str(source[0]), 'rU').read()
         patterns = open(str(source[1]))
         for pattern in patterns:
-            if pattern.strip() and not re.search(pattern.strip(), data, re.MULTILINE):
+            if pattern.strip():
+                invert = False
+                if pattern.startswith('! '):
+                    invert = True
+                    pattern = pattern[2:]
+                
+                status = re.search(pattern.strip(), data, re.MULTILINE)
+                
+                if not status and not invert:
                     print '\033[31m[FAIL]\033[0m   Pattern not found in ' + str(source[0]) + ': ' + pattern
                     return 1
+                elif status and invert:
+                    print '\033[31m[FAIL]\033[0m   Pattern should not exist, but does in ' + str(source[0]) + ': ' + pattern
+                    return 1
         else:
             print '\033[32m[ OK ]\033[0m   All patterns found in ' + str(source[0])
             return 0

tests/splint/SConscript

@@ -0,0 +1,16 @@
+# Check the nanopb core using splint
+
+Import('env')
+
+p = env.WhereIs('splint')
+
+if p:
+    env.Command('pb_decode.splint', '$NANOPB/pb_decode.c',
+        'splint -f splint/splint.rc $SOURCE 2> $TARGET')
+
+    env.Command('pb_encode.splint', '$NANOPB/pb_encode.c',
+        'splint -f splint/splint.rc $SOURCE 2> $TARGET')
+
+    env.Command('pb_common.splint', '$NANOPB/pb_common.c',
+        'splint -f splint/splint.rc $SOURCE 2> $TARGET')
+

tests/splint/splint.rc

@@ -0,0 +1,37 @@
++checks
++partial
++matchanyintegral
++strictlib
+-nullassign
+-predboolint
+-predboolptr
++ptrnegate
+-switchloopbreak
++ignoresigns
+-infloopsuncon
+-type
+
+# splint's memory checks don't quite work without annotations
+-mustfreeonly
+-compmempass
+-nullret
+-observertrans
+-statictrans
+-compdestroy
+-nullpass
+-nullstate
+-compdef
+-usereleased
+-temptrans
+-dependenttrans
+-kepttrans
+-branchstate
+-immediatetrans
+-mustfreefresh
+
+# These tests give false positives, compiler typically has
+# better warnings for these.
+-noret
+-noeffect
+-usedef
+