Publishing nanopb-0.2.4

Update issue 93
Status: FixedInGit

CHANGELOG

@@ -1,4 +1,22 @@
-nanopb-0.2.2
+nanopb-0.2.4 (2013-11-07)
+ Remove the deprecated NANOPB_INTERNALS functions from public API.
+ Document the security model.
+ Check array and bytes max sizes when encoding (issue 90)
+ Add #defines for maximum encoded message size (issue 89)
+ Add #define tags for extension fields (issue 93)
+ Fix MISRA C violations (issue 91)
+ Clean up pb_field_t definition with typedefs.
+
+nanopb-0.2.3 (2013-09-18)
+ Improve compatibility by removing ternary operator from initializations (issue 88)
+ Fix build error on Visual C++ (issue 84, patch by Markus Schwarzenberg)
+ Don't stop on unsupported extension fields (issue 83)
+ Add an example pb_syshdr.h file for non-C99 compilers
+ Reorganize tests and examples into subfolders (issue 63)
+ Switch from Makefiles to scons for building the tests
+ Make the tests buildable on Windows
+
+nanopb-0.2.2 (2013-08-18)
  Add support for extension fields (issue 17)
  Fix unknown fields in empty message (issue 78)
  Include the field tags in the generated .pb.h file.
@@ -7,7 +25,7 @@ nanopb-0.2.2
  Documentation improvements (issues 12, 77 and others)
  Improved tests
 
-nanopb-0.2.1
+nanopb-0.2.1 (2013-04-14)
  NOTE: The default callback function signature has changed.
        If you don't want to update your code, define PB_OLD_CALLBACK_STYLE.
  
@@ -24,7 +42,7 @@ nanopb-0.2.1
  Various new generator options
  Improved tests
 
-nanopb-0.2.0
+nanopb-0.2.0 (2013-03-02)
  NOTE: This release requires you to regenerate all .pb.c
        files. Files generated by older versions will not
        compile anymore.
@@ -37,7 +55,7 @@ nanopb-0.2.0
  Add option to give file extension to generator (by Michael Haberler)
  Documentation updates
 
-nanopb-0.1.9
+nanopb-0.1.9 (2013-02-13)
  Fixed error message bugs (issues 52, 56)
  Sanitize #ifndef filename (issue 50)
  Performance improvements
@@ -48,13 +66,13 @@ nanopb-0.1.9
  Added generator option to make message structs packed. (issue 49)
  Add more test cases.
 
-nanopb-0.1.8
+nanopb-0.1.8 (2012-12-13)
  Fix bugs in the enum short names introduced in 0.1.7 (issues 42, 43)
  Fix STATIC_ASSERT macro when using multiple .proto files. (issue 41)
  Fix missing initialization of istream.errmsg
  Make tests/Makefile work for non-gcc compilers (issue 40)
 
-nanopb-0.1.7
+nanopb-0.1.7 (2012-11-11)
  Remove "skip" mode from pb_istream_t callbacks. Example implementation had a bug. (issue 37)
  Add option to use shorter names for enum values (issue 38)
  Improve options support in generator (issues 12, 30)
@@ -64,19 +82,19 @@ nanopb-0.1.7
  Add buffer size check in example (issue 34)
  Fix build warnings on MS compilers (issue 33)
 
-nanopb-0.1.6
+nanopb-0.1.6 (2012-09-02)
  Reorganize the field decoder interface (issue 2)
  Improve performance in submessage decoding (issue 28)
  Implement error messages in the decoder side (issue 7)
  Extended testcases (alltypes test is now complete).
  Fix some compiler warnings (issues 25, 26, 27, 32).
 
-nanopb-0.1.5
+nanopb-0.1.5 (2012-08-04)
  Fix bug in decoder with packed arrays (issue 23).
  Extended testcases.
  Fix some compiler warnings.
 
-nanopb-0.1.4
+nanopb-0.1.4 (2012-07-05)
  Add compile-time options for easy-to-use >255 field support.
  Improve the detection of missing required fields.
  Added example on how to handle union messages.
@@ -84,20 +102,20 @@ nanopb-0.1.4
  Fix problems that stopped the code from compiling with some compilers.
  Fix some compiler warnings.
 
-nanopb-0.1.3
+nanopb-0.1.3 (2012-06-12)
  Refactor the field encoder interface.
  Improve generator error messages (issue 5)
  Add descriptor.proto into the #include exclusion list
  Fix some compiler warnings.
 
-nanopb-0.1.2
+nanopb-0.1.2 (2012-02-15)
  Make the generator to generate include for other .proto files (issue 4).
  Fixed generator not working on Windows (issue 3)
 
-nanopb-0.1.1
+nanopb-0.1.1 (2012-01-14)
  Fixed bug in encoder with 'bytes' fields (issue 1).
  Fixed a bug in the generator that caused a compiler error on sfixed32 and sfixed64 fields.
  Extended testcases.
 
-nanopb-0.1.0
+nanopb-0.1.0 (2012-01-06)
  First stable release.

docs/Makefile

@@ -1,4 +1,4 @@
-all: index.html concepts.html reference.html \
+all: index.html concepts.html reference.html security.html \
 	generator_flow.png
 
 %.png: %.svg

docs/index.rst

@@ -90,22 +90,37 @@ After that, buffer will contain the encoded message.
 The number of bytes in the message is stored in *stream.bytes_written*.
 You can feed the message to *protoc --decode=Example message.proto* to verify its validity.
 
-For complete examples of the simple cases, see *tests/test_decode1.c* and *tests/test_encode1.c*. For an example with network interface, see the *example* subdirectory.
+For a complete example of the simple case, see *example/simple.c*.
+For a more complex example with network interface, see the *example/network_server* subdirectory.
 
 Compiler requirements
 =====================
-Nanopb should compile with most ansi-C compatible compilers. It however requires a few header files to be available:
+Nanopb should compile with most ansi-C compatible compilers. It however
+requires a few header files to be available:
 
 #) *string.h*, with these functions: *strlen*, *memcpy*, *memset*
 #) *stdint.h*, for definitions of *int32_t* etc.
 #) *stddef.h*, for definition of *size_t*
 #) *stdbool.h*, for definition of *bool*
 
-If these header files do not come with your compiler, you should be able to find suitable replacements online. Mostly the requirements are very simple, just a few basic functions and typedefs.
+If these header files do not come with your compiler, you can use the
+file *compat/pb_syshdr.h* instead. It contains an example of how to provide
+the dependencies. You may have to edit it a bit to suit your custom platform.
 
-Alternatively, you can define *PB_SYSTEM_HEADER*, which should be the name of a single header file including all the necessary definitions.
+To use the pb_syshdr.h, define *PB_SYSTEM_HEADER* to be the name of your custom
+header file. It should provide all the dependencies listed above.
 
-Debugging and testing
+Running the test cases
-=====================
+======================
-Extensive unittests are included under the *tests* folder. Just type *make* there to run the tests.
+Extensive unittests and test cases are included under the *tests* folder.
+
+To build the tests, you will need the `scons`__ build system. The tests should
+be runnable on most platforms. Windows and Linux builds are regularly tested.
+
+__ http://www.scons.org/
+
+In addition to the build system, you will also need a working Google Protocol
+Buffers *protoc* compiler, and the Python bindings for Protocol Buffers. On
+Debian-based systems, install the following packages: *protobuf-compiler*,
+*python-protobuf* and *libprotobuf-dev*.
 

docs/menu.rst

@@ -3,8 +3,9 @@
     1) `Overview`_
     2) `Concepts`_
     3) `API reference`_
+    4) `Security model`_
     
 .. _`Overview`: index.html
 .. _`Concepts`: concepts.html
 .. _`API reference`: reference.html
-
+.. _`Security model`: security.html

docs/reference.rst

@@ -25,7 +25,9 @@ __BIG_ENDIAN__                 Set this if your platform stores integers and
                                systems (different layout for ints and floats)
                                are currently not supported.
 NANOPB_INTERNALS               Set this to expose the field encoder functions
-                               that are hidden since nanopb-0.1.3.
+                               that are hidden since nanopb-0.1.3. Starting
+                               with nanopb-0.2.4, this flag does nothing. Use
+                               the newer functions that have better interface.
 PB_MAX_REQUIRED_FIELDS         Maximum number of required fields to check for
                                presence. Default value is 64. Increases stack
                                usage 1 byte per every 8 fields. Compiler

docs/security.rst

@@ -0,0 +1,79 @@
+======================
+Nanopb: Security model
+======================
+
+.. include :: menu.rst
+
+.. contents ::
+
+
+
+Importance of security in a Protocol Buffers library
+====================================================
+In the context of protocol buffers, security comes into play when decoding
+untrusted data. Naturally, if the attacker can modify the contents of a
+protocol buffers message, he can feed the application any values possible.
+Therefore the application itself must be prepared to receive untrusted values.
+
+Where nanopb plays a part is preventing the attacker from running arbitrary
+code on the target system. Mostly this means that there must not be any
+possibility to cause buffer overruns, memory corruption or invalid pointers
+by the means of crafting a malicious message.
+
+Division of trusted and untrusted data
+======================================
+The following data is regarded as **trusted**. It must be under the control of
+the application writer. Malicious data in these structures could cause
+security issues, such as execution of arbitrary code:
+
+1. Callback and extension fields in message structures given to pb_encode()
+   and pb_decode(). These fields are memory pointers, and are generated
+   depending on the .proto file.
+2. The automatically generated field definitions, i.e. *pb_field_t* lists.
+3. Contents of the *pb_istream_t* and *pb_ostream_t* structures (this does not
+   mean the contents of the stream itself, just the stream definition).
+
+The following data is regarded as **untrusted**. Invalid/malicious data in
+these will cause "garbage in, garbage out" behaviour. It will not cause
+buffer overflows, information disclosure or other security problems:
+
+1. All data read from *pb_istream_t*.
+2. All fields in message structures, except callbacks and extensions.
+   (Beginning with nanopb-0.2.4, in earlier versions the field sizes are partially unchecked.)
+
+Invariants
+==========
+The following invariants are maintained during operation, even if the
+untrusted data has been maliciously crafted:
+
+1. Nanopb will never read more than *bytes_left* bytes from *pb_istream_t*.
+2. Nanopb will never write more than *max_size* bytes to *pb_ostream_t*.
+3. Nanopb will never access memory out of bounds of the message structure.
+4. After pb_decode() returns successfully, the message structure will be
+   internally consistent:
+
+   - The *count* fields of arrays will not exceed the array size.
+   - The *size* field of bytes will not exceed the allocated size.
+   - All string fields will have null terminator.
+
+5. After pb_encode() returns successfully, the resulting message is a valid
+   protocol buffers message. (Except if user-defined callbacks write incorrect
+   data.)
+
+Further considerations
+======================
+Even if the nanopb library is free of any security issues, there are still
+several possible attack vectors that the application author must consider.
+The following list is not comprehensive:
+
+1. Stack usage may depend on the contents of the message. The message
+   definition places an upper bound on how much stack will be used. Tests
+   should be run with all fields present, to record the maximum possible
+   stack usage.
+2. Callbacks can do anything. The code for the callbacks must be carefully
+   checked if they are used with untrusted data.
+3. If using stream input, a maximum size should be set in *pb_istream_t* to
+   stop a denial of service attack from using an infinite message.
+4. If using network sockets as streams, a timeout should be set to stop
+   denial of service attacks.
+

example/Makefile

@@ -1,14 +0,0 @@
-CFLAGS=-ansi -Wall -Werror -I .. -g -O0
-DEPS=../pb_decode.c ../pb_decode.h ../pb_encode.c ../pb_encode.h ../pb.h
-
-all: server client
-
-clean:
-	rm -f server client fileproto.pb.c fileproto.pb.h
-
-%: %.c $(DEPS) fileproto.pb.h fileproto.pb.c
-	$(CC) $(CFLAGS) -o $@ $< ../pb_decode.c ../pb_encode.c fileproto.pb.c common.c
-
-fileproto.pb.c fileproto.pb.h: fileproto.proto ../generator/nanopb_generator.py
-	protoc -I. -I../generator -I/usr/include -ofileproto.pb $<
-	python ../generator/nanopb_generator.py fileproto.pb

example_avr_double/Makefile

@@ -1,22 +0,0 @@
-CFLAGS=-Wall -Werror -I .. -g -O0
-DEPS=double_conversion.c ../pb_decode.c ../pb_decode.h ../pb_encode.c ../pb_encode.h ../pb.h
-
-all: run_tests
-
-clean:
-	rm -f test_conversions encode_double decode_double doubleproto.pb.c doubleproto.pb.h
-
-test_conversions: test_conversions.c double_conversion.c
-	$(CC) $(CFLAGS) -o $@ $^
-
-%: %.c $(DEPS) doubleproto.pb.h doubleproto.pb.c
-	$(CC) $(CFLAGS) -o $@ $< double_conversion.c ../pb_decode.c ../pb_encode.c doubleproto.pb.c
-
-doubleproto.pb.c doubleproto.pb.h: doubleproto.proto ../generator/nanopb_generator.py
-	protoc -I. -I../generator -I/usr/include -odoubleproto.pb $<
-	python ../generator/nanopb_generator.py doubleproto.pb
-
-run_tests: test_conversions encode_double decode_double
-	./test_conversions
-	./encode_double | ./decode_double
-    

example_unions/Makefile

@@ -1,17 +0,0 @@
-CFLAGS=-ansi -Wall -Werror -I .. -g -O0
-DEPS=../pb_decode.c ../pb_decode.h ../pb_encode.c ../pb_encode.h ../pb.h
-
-all: encode decode
-	./encode 1 | ./decode
-	./encode 2 | ./decode
-	./encode 3 | ./decode
-
-clean:
-	rm -f encode unionproto.pb.h unionproto.pb.c
-
-%: %.c $(DEPS) unionproto.pb.h unionproto.pb.c
-	$(CC) $(CFLAGS) -o $@ $< ../pb_decode.c ../pb_encode.c unionproto.pb.c
-
-unionproto.pb.h unionproto.pb.c: unionproto.proto ../generator/nanopb_generator.py
-	protoc -I. -I../generator -I/usr/include -ounionproto.pb $<
-	python ../generator/nanopb_generator.py unionproto.pb

examples/network_server/Makefile

@@ -0,0 +1,19 @@
+CFLAGS = -ansi -Wall -Werror -g -O0
+
+# Path to the nanopb root folder
+NANOPB_DIR = ../..
+DEPS = $(NANOPB_DIR)/pb_decode.c $(NANOPB_DIR)/pb_decode.h \
+       $(NANOPB_DIR)/pb_encode.c $(NANOPB_DIR)/pb_encode.h $(NANOPB_DIR)/pb.h
+CFLAGS += -I$(NANOPB_DIR)
+
+all: server client
+
+clean:
+	rm -f server client fileproto.pb.c fileproto.pb.h
+
+%: %.c $(DEPS) fileproto.pb.h fileproto.pb.c
+	$(CC) $(CFLAGS) -o $@ $< $(NANOPB_DIR)/pb_decode.c $(NANOPB_DIR)/pb_encode.c fileproto.pb.c common.c
+
+fileproto.pb.c fileproto.pb.h: fileproto.proto $(NANOPB_DIR)/generator/nanopb_generator.py
+	protoc -ofileproto.pb $<
+	python $(NANOPB_DIR)/generator/nanopb_generator.py fileproto.pb

examples/network_server/README

@@ -0,0 +1,60 @@
+Nanopb example "network_server"
+===============================
+
+This example demonstrates the use of nanopb to communicate over network
+connections. It consists of a server that sends file listings, and of
+a client that requests the file list from the server.
+
+Example usage
+-------------
+
+user@host:~/nanopb/examples/network_server$ make        # Build the example
+protoc -ofileproto.pb fileproto.proto
+python ../../generator/nanopb_generator.py fileproto.pb
+Writing to fileproto.pb.h and fileproto.pb.c
+cc -ansi -Wall -Werror -I .. -g -O0 -I../.. -o server server.c
+    ../../pb_decode.c ../../pb_encode.c fileproto.pb.c common.c
+cc -ansi -Wall -Werror -I .. -g -O0 -I../.. -o client client.c
+    ../../pb_decode.c ../../pb_encode.c fileproto.pb.c common.c
+
+user@host:~/nanopb/examples/network_server$ ./server &  # Start the server on background
+[1] 24462
+
+petteri@oddish:~/nanopb/examples/network_server$ ./client /bin  # Request the server to list /bin
+Got connection.
+Listing directory: /bin
+1327119    bzdiff
+1327126    bzless
+1327147    ps
+1327178    ntfsmove
+1327271    mv
+1327187    mount
+1327259    false
+1327266    tempfile
+1327285    zfgrep
+1327165    gzexe
+1327204    nc.openbsd
+1327260    uname
+
+
+Details of implementation
+-------------------------
+fileproto.proto contains the portable Google Protocol Buffers protocol definition.
+It could be used as-is to implement a server or a client in any other language, for
+example Python or Java.
+
+fileproto.options contains the nanopb-specific options for the protocol file. This
+sets the amount of space allocated for file names when decoding messages.
+
+common.c/h contains functions that allow nanopb to read and write directly from
+network socket. This way there is no need to allocate a separate buffer to store
+the message.
+
+server.c contains the code to open a listening socket, to respond to clients and
+to list directory contents.
+
+client.c contains the code to connect to a server, to send a request and to print
+the response message.
+
+The code is implemented using the POSIX socket api, but it should be easy enough
+to port into any other socket api, such as lwip.

examples/simple/Makefile

@@ -0,0 +1,22 @@
+# Compiler flags to enable all warnings & debug info
+CFLAGS = -Wall -Werror -g -O0
+
+# Path to the nanopb root folder
+NANOPB_DIR = ../..
+CFLAGS += -I$(NANOPB_DIR)
+
+# C source code files that are required
+CSRC  = simple.c                   # The main program
+CSRC += simple.pb.c                # The compiled protocol definition
+CSRC += $(NANOPB_DIR)/pb_encode.c  # The nanopb encoder
+CSRC += $(NANOPB_DIR)/pb_decode.c  # The nanopb decoder
+
+# Build rule for the main program
+simple: $(CSRC)
+	$(CC) $(CFLAGS) -osimple $(CSRC)
+
+# Build rule for the protocol
+simple.pb.c: simple.proto
+	protoc -osimple.pb simple.proto
+	python $(NANOPB_DIR)/generator/nanopb_generator.py simple.pb
+

examples/simple/README

@@ -0,0 +1,30 @@
+Nanopb example "simple"
+=======================
+
+This example demonstrates the very basic use of nanopb. It encodes and
+decodes a simple message.
+
+The code uses four different API functions:
+
+  * pb_ostream_from_buffer() to declare the output buffer that is to be used
+  * pb_encode() to encode a message
+  * pb_istream_from_buffer() to declare the input buffer that is to be used
+  * pb_decode() to decode a message
+
+Example usage
+-------------
+
+On Linux, simply type "make" to build the example. After that, you can
+run it with the command: ./simple
+
+On other platforms, you first have to compile the protocol definition using
+the following two commands::
+
+  protoc -osimple.pb simple.proto
+  python nanopb_generator.py simple.pb
+
+After that, add the following four files to your project and compile:
+
+  simple.c  simple.pb.c  pb_encode.c  pb_decode.c
+
+

examples/simple/simple.c

@@ -0,0 +1,68 @@
+#include <stdio.h>
+#include <pb_encode.h>
+#include <pb_decode.h>
+#include "simple.pb.h"
+
+int main()
+{
+    /* This is the buffer where we will store our message. */
+    uint8_t buffer[128];
+    size_t message_length;
+    bool status;
+    
+    /* Encode our message */
+    {
+        /* Allocate space on the stack to store the message data.
+         *
+         * Nanopb generates simple struct definitions for all the messages.
+         * - check out the contents of simple.pb.h! */
+        SimpleMessage message;
+        
+        /* Create a stream that will write to our buffer. */
+        pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
+        
+        /* Fill in the lucky number */
+        message.lucky_number = 13;
+        
+        /* Now we are ready to encode the message! */
+        status = pb_encode(&stream, SimpleMessage_fields, &message);
+        message_length = stream.bytes_written;
+        
+        /* Then just check for any errors.. */
+        if (!status)
+        {
+            printf("Encoding failed: %s\n", PB_GET_ERROR(&stream));
+            return 1;
+        }
+    }
+    
+    /* Now we could transmit the message over network, store it in a file or
+     * wrap it to a pigeon's leg.
+     */
+
+    /* But because we are lazy, we will just decode it immediately. */
+    
+    {
+        /* Allocate space for the decoded message. */
+        SimpleMessage message;
+        
+        /* Create a stream that reads from the buffer. */
+        pb_istream_t stream = pb_istream_from_buffer(buffer, message_length);
+        
+        /* Now we are ready to decode the message. */
+        status = pb_decode(&stream, SimpleMessage_fields, &message);
+        
+        /* Check for errors... */
+        if (!status)
+        {
+            printf("Decoding failed: %s\n", PB_GET_ERROR(&stream));
+            return 1;
+        }
+        
+        /* Print the data contained in the message. */
+        printf("Your lucky number was %d!\n", message.lucky_number);
+    }
+    
+    return 0;
+}
+

examples/simple/simple.proto

@@ -0,0 +1,7 @@
+// A very simple protocol definition, consisting of only
+// one message.
+
+message SimpleMessage {
+    required int32 lucky_number = 1;
+}
+

examples/using_double_on_avr/Makefile

@@ -0,0 +1,29 @@
+CFLAGS = -Wall -Werror -g -O0
+
+# Path to the nanopb root directory
+NANOPB_DIR = ../..
+DEPS = double_conversion.c $(NANOPB_DIR)/pb.h \
+	$(NANOPB_DIR)/pb_decode.c $(NANOPB_DIR)/pb_decode.h \
+	$(NANOPB_DIR)/pb_encode.c $(NANOPB_DIR)/pb_encode.h
+CFLAGS += -I$(NANOPB_DIR)
+
+all: run_tests
+
+clean:
+	rm -f test_conversions encode_double decode_double doubleproto.pb.c doubleproto.pb.h
+
+test_conversions: test_conversions.c double_conversion.c
+	$(CC) $(CFLAGS) -o $@ $^
+
+%: %.c $(DEPS) doubleproto.pb.h doubleproto.pb.c
+	$(CC) $(CFLAGS) -o $@ $< double_conversion.c \
+		$(NANOPB_DIR)/pb_decode.c $(NANOPB_DIR)/pb_encode.c doubleproto.pb.c
+
+doubleproto.pb.c doubleproto.pb.h: doubleproto.proto $(NANOPB_DIR)/generator/nanopb_generator.py
+	protoc -odoubleproto.pb $<
+	python $(NANOPB_DIR)/generator/nanopb_generator.py doubleproto.pb
+
+run_tests: test_conversions encode_double decode_double
+	./test_conversions
+	./encode_double | ./decode_double
+    

examples/using_double_on_avr/README

@@ -1,3 +1,6 @@
+Nanopb example "using_double_on_avr"
+====================================
+
 Some processors/compilers, such as AVR-GCC, do not support the double
 datatype. Instead, they have sizeof(double) == 4. Because protocol
 binary format uses the double encoding directly, this causes trouble
@@ -9,7 +12,7 @@ platforms. The file double_conversion.c provides functions that
 convert these values to/from floats, without relying on compiler
 support.
 
-To use this method, you need to make two modifications to your code:
+To use this method, you need to make some modifications to your code:
 
 1) Change all 'double' fields into 'fixed64' in the .proto.
 
@@ -17,6 +20,6 @@ To use this method, you need to make two modifications to your code:
 
 3) Whenever reading a 'double' field, use double_to_float().
 
-The conversion routines should be as accurate as the float datatype can
+The conversion routines are as accurate as the float datatype can
 be. Furthermore, they should handle all special values (NaN, inf, denormalized
 numbers) correctly. There are testcases in test_conversions.c.

examples/using_union_messages/Makefile

@@ -0,0 +1,22 @@
+CFLAGS = -ansi -Wall -Werror -g -O0
+
+# Path to the nanopb root folder
+NANOPB_DIR = ../..
+DEPS = $(NANOPB_DIR)/pb_decode.c $(NANOPB_DIR)/pb_decode.h \
+       $(NANOPB_DIR)/pb_encode.c $(NANOPB_DIR)/pb_encode.h $(NANOPB_DIR)/pb.h
+CFLAGS += -I$(NANOPB_DIR)
+
+all: encode decode
+	./encode 1 | ./decode
+	./encode 2 | ./decode
+	./encode 3 | ./decode
+
+clean:
+	rm -f encode unionproto.pb.h unionproto.pb.c
+
+%: %.c $(DEPS) unionproto.pb.h unionproto.pb.c
+	$(CC) $(CFLAGS) -o $@ $< $(NANOPB_DIR)/pb_decode.c $(NANOPB_DIR)/pb_encode.c unionproto.pb.c
+
+unionproto.pb.h unionproto.pb.c: unionproto.proto $(NANOPB_DIR)/generator/nanopb_generator.py
+	protoc -ounionproto.pb $<
+	python $(NANOPB_DIR)/generator/nanopb_generator.py unionproto.pb

examples/using_union_messages/README

@@ -0,0 +1,52 @@
+Nanopb example "using_union_messages"
+=====================================
+
+Union messages is a common technique in Google Protocol Buffers used to
+represent a group of messages, only one of which is passed at a time.
+It is described in Google's documentation:
+https://developers.google.com/protocol-buffers/docs/techniques#union
+
+This directory contains an example on how to encode and decode union messages
+with minimal memory usage. Usually, nanopb would allocate space to store
+all of the possible messages at the same time, even though at most one of
+them will be used at a time.
+
+By using some of the lower level nanopb APIs, we can manually generate the
+top level message, so that we only need to allocate the one submessage that
+we actually want. Similarly when decoding, we can manually read the tag of
+the top level message, and only then allocate the memory for the submessage
+after we already know its type.
+
+
+Example usage
+-------------
+
+Type `make` to run the example. It will build it and run commands like
+following:
+
+./encode 1 | ./decode
+Got MsgType1: 42
+./encode 2 | ./decode
+Got MsgType2: true
+./encode 3 | ./decode
+Got MsgType3: 3 1415
+
+This simply demonstrates that the "decode" program has correctly identified
+the type of the received message, and managed to decode it.
+
+
+Details of implementation
+-------------------------
+
+unionproto.proto contains the protocol used in the example. It consists of
+three messages: MsgType1, MsgType2 and MsgType3, which are collected together
+into UnionMessage.
+
+encode.c takes one command line argument, which should be a number 1-3. It
+then fills in and encodes the corresponding message, and writes it to stdout.
+
+decode.c reads a UnionMessage from stdin. Then it calls the function
+decode_unionmessage_type() to determine the type of the message. After that,
+the corresponding message is decoded and the contents of it printed to the
+screen.
+

generator/Makefile

@@ -1,2 +1,5 @@
 nanopb_pb2.py: nanopb.proto
 	protoc --python_out=. -I /usr/include -I . nanopb.proto
+
+plugin_pb2.py: plugin.proto
+	protoc --python_out=. -I /usr/include -I . plugin.proto

generator/nanopb_generator.py

@@ -1,5 +1,7 @@
+#!/usr/bin/python
+
 '''Generate header file for nanopb from a ProtoBuf FileDescriptorSet.'''
-nanopb_version = "nanopb-0.2.3-dev"
+nanopb_version = "nanopb-0.2.4"
 
 try:
     import google.protobuf.descriptor_pb2 as descriptor
@@ -36,22 +38,22 @@ except:
 import time
 import os.path
 
-# Values are tuple (c type, pb type)
+# Values are tuple (c type, pb type, encoded size)
 FieldD = descriptor.FieldDescriptorProto
 datatypes = {
-    FieldD.TYPE_BOOL:       ('bool',     'BOOL'),
+    FieldD.TYPE_BOOL:       ('bool',     'BOOL',        1),
-    FieldD.TYPE_DOUBLE:     ('double',   'DOUBLE'),
+    FieldD.TYPE_DOUBLE:     ('double',   'DOUBLE',      8),
-    FieldD.TYPE_FIXED32:    ('uint32_t', 'FIXED32'),
+    FieldD.TYPE_FIXED32:    ('uint32_t', 'FIXED32',     4),
-    FieldD.TYPE_FIXED64:    ('uint64_t', 'FIXED64'),
+    FieldD.TYPE_FIXED64:    ('uint64_t', 'FIXED64',     8),
-    FieldD.TYPE_FLOAT:      ('float',    'FLOAT'),
+    FieldD.TYPE_FLOAT:      ('float',    'FLOAT',       4),
-    FieldD.TYPE_INT32:      ('int32_t',  'INT32'),
+    FieldD.TYPE_INT32:      ('int32_t',  'INT32',       5),
-    FieldD.TYPE_INT64:      ('int64_t',  'INT64'),
+    FieldD.TYPE_INT64:      ('int64_t',  'INT64',      10),
-    FieldD.TYPE_SFIXED32:   ('int32_t',  'SFIXED32'),
+    FieldD.TYPE_SFIXED32:   ('int32_t',  'SFIXED32',    4),
-    FieldD.TYPE_SFIXED64:   ('int64_t',  'SFIXED64'),
+    FieldD.TYPE_SFIXED64:   ('int64_t',  'SFIXED64',    8),
-    FieldD.TYPE_SINT32:     ('int32_t',  'SINT32'),
+    FieldD.TYPE_SINT32:     ('int32_t',  'SINT32',      5),
-    FieldD.TYPE_SINT64:     ('int64_t',  'SINT64'),
+    FieldD.TYPE_SINT64:     ('int64_t',  'SINT64',     10),
-    FieldD.TYPE_UINT32:     ('uint32_t', 'UINT32'),
+    FieldD.TYPE_UINT32:     ('uint32_t', 'UINT32',      5),
-    FieldD.TYPE_UINT64:     ('uint64_t', 'UINT64')
+    FieldD.TYPE_UINT64:     ('uint64_t', 'UINT64',     10)
 }
 
 class Names:
@@ -81,6 +83,55 @@ def names_from_type_name(type_name):
         raise NotImplementedError("Lookup of non-absolute type names is not supported")
     return Names(type_name[1:].split('.'))
 
+def varint_max_size(max_value):
+    '''Returns the maximum number of bytes a varint can take when encoded.'''
+    for i in range(1, 11):
+        if (max_value >> (i * 7)) == 0:
+            return i
+    raise ValueError("Value too large for varint: " + str(max_value))
+
+assert varint_max_size(0) == 1
+assert varint_max_size(127) == 1
+assert varint_max_size(128) == 2
+
+class EncodedSize:
+    '''Class used to represent the encoded size of a field or a message.
+    Consists of a combination of symbolic sizes and integer sizes.'''
+    def __init__(self, value = 0, symbols = []):
+        if isinstance(value, (str, Names)):
+            symbols = [str(value)]
+            value = 0
+        self.value = value
+        self.symbols = symbols
+    
+    def __add__(self, other):
+        if isinstance(other, (int, long)):
+            return EncodedSize(self.value + other, self.symbols)
+        elif isinstance(other, (str, Names)):
+            return EncodedSize(self.value, self.symbols + [str(other)])
+        elif isinstance(other, EncodedSize):
+            return EncodedSize(self.value + other.value, self.symbols + other.symbols)
+        else:
+            raise ValueError("Cannot add size: " + repr(other))
+
+    def __mul__(self, other):
+        if isinstance(other, (int, long)):
+            return EncodedSize(self.value * other, [str(other) + '*' + s for s in self.symbols])
+        else:
+            raise ValueError("Cannot multiply size: " + repr(other))
+
+    def __str__(self):
+        if not self.symbols:
+            return str(self.value)
+        else:
+            return '(' + str(self.value) + ' + ' + ' + '.join(self.symbols) + ')'
+
+    def upperlimit(self):
+        if not self.symbols:
+            return self.value
+        else:
+            return 2**32 - 1
+
 class Enum:
     def __init__(self, names, desc, enum_options):
         '''desc is EnumDescriptorProto'''
@@ -111,6 +162,7 @@ class Field:
         self.max_size = None
         self.max_count = None
         self.array_decl = ""
+        self.enc_size = None
         
         # Parse field options
         if field_options.HasField("max_size"):
@@ -139,12 +191,13 @@ class Field:
         
         # Decide the C data type to use in the struct.
         if datatypes.has_key(desc.type):
-            self.ctype, self.pbtype = datatypes[desc.type]
+            self.ctype, self.pbtype, self.enc_size = datatypes[desc.type]
         elif desc.type == FieldD.TYPE_ENUM:
             self.pbtype = 'ENUM'
             self.ctype = names_from_type_name(desc.type_name)
             if self.default is not None:
                 self.default = self.ctype + self.default
+            self.enc_size = 5 # protoc rejects enum values > 32 bits
         elif desc.type == FieldD.TYPE_STRING:
             self.pbtype = 'STRING'
             if self.max_size is None:
@@ -152,15 +205,18 @@ class Field:
             else:
                 self.ctype = 'char'
                 self.array_decl += '[%d]' % self.max_size
+                self.enc_size = varint_max_size(self.max_size) + self.max_size
         elif desc.type == FieldD.TYPE_BYTES:
             self.pbtype = 'BYTES'
             if self.max_size is None:
                 can_be_static = False
             else:
                 self.ctype = self.struct_name + self.name + 't'
+                self.enc_size = varint_max_size(self.max_size) + self.max_size
         elif desc.type == FieldD.TYPE_MESSAGE:
             self.pbtype = 'MESSAGE'
             self.ctype = self.submsgname = names_from_type_name(desc.type_name)
+            self.enc_size = None # Needs to be filled in after the message type is available
         else:
             raise NotImplementedError(desc.type)
         
@@ -244,10 +300,11 @@ class Field:
         '''Return the pb_field_t initializer to use in the constant array.
         prev_field_name is the name of the previous field or None.
         '''
-        result  = '    PB_FIELD(%3d, ' % self.tag
+        result  = '    PB_FIELD2(%3d, ' % self.tag
         result += '%-8s, ' % self.pbtype
         result += '%s, ' % self.rules
         result += '%s, ' % self.allocation
+        result += '%s, ' % ("FIRST" if not prev_field_name else "OTHER")
         result += '%s, ' % self.struct_name
         result += '%s, ' % self.name
         result += '%s, ' % (prev_field_name or self.name)
@@ -274,6 +331,51 @@ class Field:
 
         return max(self.tag, self.max_size, self.max_count)        
 
+    def encoded_size(self, allmsgs):
+        '''Return the maximum size that this field can take when encoded,
+        including the field tag. If the size cannot be determined, returns
+        None.'''
+        
+        if self.allocation != 'STATIC':
+            return None
+        
+        if self.pbtype == 'MESSAGE':
+            for msg in allmsgs:
+                if msg.name == self.submsgname:
+                    encsize = msg.encoded_size(allmsgs)
+                    if encsize is None:
+                        return None # Submessage size is indeterminate
+                        
+                    # Include submessage length prefix
+                    encsize += varint_max_size(encsize.upperlimit())
+                    break
+            else:
+                # Submessage cannot be found, this currently occurs when
+                # the submessage type is defined in a different file.
+                # Instead of direct numeric value, reference the size that
+                # has been #defined in the other file.
+                encsize = EncodedSize(self.submsgname + 'size')
+
+                # We will have to make a conservative assumption on the length
+                # prefix size, though.
+                encsize += 5
+
+        elif self.enc_size is None:
+            raise RuntimeError("Could not determine encoded size for %s.%s"
+                               % (self.struct_name, self.name))
+        else:
+            encsize = EncodedSize(self.enc_size)
+        
+        encsize += varint_max_size(self.tag << 3) # Tag + wire type
+
+        if self.rules == 'REPEATED':
+            # Decoders must be always able to handle unpacked arrays.
+            # Therefore we have to reserve space for it, even though
+            # we emit packed arrays ourselves.
+            encsize *= self.max_count
+        
+        return encsize
+
 
 class ExtensionRange(Field):
     def __init__(self, struct_name, range_start, field_options):
@@ -303,6 +405,12 @@ class ExtensionRange(Field):
     def tags(self):
         return ''
         
+    def encoded_size(self, allmsgs):
+        # We exclude extensions from the count, because they cannot be known
+        # until runtime. Other option would be to return None here, but this
+        # way the value remains useful if extensions are not used.
+        return EncodedSize(0)
+
 class ExtensionField(Field):
     def __init__(self, struct_name, desc, field_options):
         self.fullname = struct_name + desc.name
@@ -315,6 +423,11 @@ class ExtensionField(Field):
             self.skip = False
             self.rules = 'OPTEXT'
 
+    def tags(self):
+        '''Return the #define for the tag number of this field.'''
+        identifier = '%s_tag' % self.fullname
+        return '#define %-40s %d\n' % (identifier, self.tag)
+
     def extension_decl(self):
         '''Declaration of the extension type in the .pb.h file'''
         if self.skip:
@@ -426,6 +539,18 @@ class Message:
         result += '    PB_LAST_FIELD\n};'
         return result
 
+    def encoded_size(self, allmsgs):
+        '''Return the maximum size that this message can take when encoded.
+        If the size cannot be determined, returns None.
+        '''
+        size = EncodedSize(0)
+        for field in self.fields:
+            fsize = field.encoded_size(allmsgs)
+            if fsize is None:
+                return None
+            size += fsize
+        
+        return size
 
 
 # ---------------------------------------------------------------------------
@@ -589,20 +714,31 @@ def generate_header(dependencies, headername, enums, messages, extensions, optio
     for msg in sort_dependencies(messages):
         for field in msg.fields:
             yield field.tags()
+    for extension in extensions:
+        yield extension.tags()
     yield '\n'
     
     yield '/* Struct field encoding specification for nanopb */\n'
     for msg in messages:
         yield msg.fields_declaration() + '\n'
+    yield '\n'
     
-    yield '\n#ifdef __cplusplus\n'
+    yield '/* Maximum encoded size of messages (where known) */\n'
+    for msg in messages:
+        msize = msg.encoded_size(messages)
+        if msize is not None:
+            identifier = '%s_size' % msg.name
+            yield '#define %-40s %s\n' % (identifier, msize)
+    yield '\n'
+    
+    yield '#ifdef __cplusplus\n'
     yield '} /* extern "C" */\n'
     yield '#endif\n'
     
     # End of header
     yield '\n#endif\n'
 
-def generate_source(headername, enums, messages, extensions):
+def generate_source(headername, enums, messages, extensions, options):
     '''Generate content for a source file.'''
     
     yield '/* Automatically generated nanopb constant definitions */\n'
@@ -780,73 +916,126 @@ optparser.add_option("-v", "--verbose", dest="verbose", action="store_true", def
 optparser.add_option("-s", dest="settings", metavar="OPTION:VALUE", action="append", default=[],
     help="Set generator option (max_size, max_count etc.).")
 
-def process(filenames, options):
+def process_file(filename, fdesc, options):
-    '''Process the files given on the command line.'''
+    '''Process a single file.
+    filename: The full path to the .proto or .pb source file, as string.
+    fdesc: The loaded FileDescriptorSet, or None to read from the input file.
+    options: Command line options as they come from OptionsParser.
+    
+    Returns a dict:
+        {'headername': Name of header file,
+         'headerdata': Data for the .h header file,
+         'sourcename': Name of the source code file,
+         'sourcedata': Data for the .c source code file
+        }
+    '''
+    toplevel_options = nanopb_pb2.NanoPBOptions()
+    for s in options.settings:
+        text_format.Merge(s, toplevel_options)
+    
+    if not fdesc:
+        data = open(filename, 'rb').read()
+        fdesc = descriptor.FileDescriptorSet.FromString(data).file[0]
+    
+    # Check if there is a separate .options file
+    try:
+        optfilename = options.options_file % os.path.splitext(filename)[0]
+    except TypeError:
+        # No %s specified, use the filename as-is
+        optfilename = options.options_file
+    
+    if options.verbose:
+        print 'Reading options from ' + optfilename
+    
+    if os.path.isfile(optfilename):
+        Globals.separate_options = read_options_file(open(optfilename, "rU"))
+    else:
+        Globals.separate_options = []
+    
+    # Parse the file
+    file_options = get_nanopb_suboptions(fdesc, toplevel_options, Names([filename]))
+    enums, messages, extensions = parse_file(fdesc, file_options)
+    
+    # Decide the file names
+    noext = os.path.splitext(filename)[0]
+    headername = noext + '.' + options.extension + '.h'
+    sourcename = noext + '.' + options.extension + '.c'
+    headerbasename = os.path.basename(headername)
+    
+    # List of .proto files that should not be included in the C header file
+    # even if they are mentioned in the source .proto.
+    excludes = ['nanopb.proto', 'google/protobuf/descriptor.proto'] + options.exclude
+    dependencies = [d for d in fdesc.dependency if d not in excludes]
+    
+    headerdata = ''.join(generate_header(dependencies, headerbasename, enums,
+                                         messages, extensions, options))
+
+    sourcedata = ''.join(generate_source(headerbasename, enums,
+                                         messages, extensions, options))
+
+    return {'headername': headername, 'headerdata': headerdata,
+            'sourcename': sourcename, 'sourcedata': sourcedata}
+    
+def main_cli():
+    '''Main function when invoked directly from the command line.'''
+    
+    options, filenames = optparser.parse_args()
     
     if not filenames:
         optparser.print_help()
-        return False
+        sys.exit(1)
     
     if options.quiet:
         options.verbose = False
 
     Globals.verbose_options = options.verbose
     
-    toplevel_options = nanopb_pb2.NanoPBOptions()
-    for s in options.settings:
-        text_format.Merge(s, toplevel_options)
-    
     for filename in filenames:
-        data = open(filename, 'rb').read()
+        results = process_file(filename, None, options)
-        fdesc = descriptor.FileDescriptorSet.FromString(data)
-        
-        # Check if any separate options are specified
-        try:
-            optfilename = options.options_file % os.path.splitext(filename)[0]
-        except TypeError:
-            # No %s specified, use the filename as-is
-            optfilename = options.options_file
-        
-        if options.verbose:
-            print 'Reading options from ' + optfilename
-        
-        if os.path.isfile(optfilename):
-            Globals.separate_options = read_options_file(open(optfilename, "rU"))
-        else:
-            Globals.separate_options = []
-        
-        # Parse the file
-        file_options = get_nanopb_suboptions(fdesc.file[0], toplevel_options, Names([filename]))
-        enums, messages, extensions = parse_file(fdesc.file[0], file_options)
-        
-        noext = os.path.splitext(filename)[0]
-        headername = noext + '.' + options.extension + '.h'
-        sourcename = noext + '.' + options.extension + '.c'
-        headerbasename = os.path.basename(headername)
         
         if not options.quiet:
-            print "Writing to " + headername + " and " + sourcename
+            print "Writing to " + results['headername'] + " and " + results['sourcename']
     
-        # List of .proto files that should not be included in the C header file
+        open(results['headername'], 'w').write(results['headerdata'])
-        # even if they are mentioned in the source .proto.
+        open(results['sourcename'], 'w').write(results['sourcedata'])        
-        excludes = ['nanopb.proto', 'google/protobuf/descriptor.proto'] + options.exclude
-        dependencies = [d for d in fdesc.file[0].dependency if d not in excludes]
 
-        header = open(headername, 'w')
+def main_plugin():
-        for part in generate_header(dependencies, headerbasename, enums,
+    '''Main function when invoked as a protoc plugin.'''
-                                    messages, extensions, options):
-            header.write(part)
 
-        source = open(sourcename, 'w')
+    import plugin_pb2
-        for part in generate_source(headerbasename, enums, messages, extensions):
+    data = sys.stdin.read()
-            source.write(part)
+    request = plugin_pb2.CodeGeneratorRequest.FromString(data)
     
-    return True
+    import shlex
+    args = shlex.split(request.parameter)
+    options, dummy = optparser.parse_args(args)
+    
+    # We can't go printing stuff to stdout
+    Globals.verbose_options = False
+    options.verbose = False
+    options.quiet = True
+    
+    response = plugin_pb2.CodeGeneratorResponse()
+    
+    for filename in request.file_to_generate:
+        for fdesc in request.proto_file:
+            if fdesc.name == filename:
+                results = process_file(filename, fdesc, options)
+                
+                f = response.file.add()
+                f.name = results['headername']
+                f.content = results['headerdata']
+
+                f = response.file.add()
+                f.name = results['sourcename']
+                f.content = results['sourcedata']    
+    
+    sys.stdout.write(response.SerializeToString())
 
 if __name__ == '__main__':
-    options, filenames = optparser.parse_args()
+    # Check if we are running as a plugin under protoc
-    status = process(filenames, options)
+    if 'protoc-gen-' in sys.argv[0]:
-    
+        main_plugin()
-    if not status:
+    else:
-        sys.exit(1)
+        main_cli()
 

generator/plugin.proto

@@ -0,0 +1,145 @@
+// Protocol Buffers - Google's data interchange format
+// Copyright 2008 Google Inc.  All rights reserved.
+// http://code.google.com/p/protobuf/
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions are
+// met:
+//
+//     * Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+//     * Redistributions in binary form must reproduce the above
+// copyright notice, this list of conditions and the following disclaimer
+// in the documentation and/or other materials provided with the
+// distribution.
+//     * Neither the name of Google Inc. nor the names of its
+// contributors may be used to endorse or promote products derived from
+// this software without specific prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+// Author: kenton@google.com (Kenton Varda)
+//
+// WARNING:  The plugin interface is currently EXPERIMENTAL and is subject to
+//   change.
+//
+// protoc (aka the Protocol Compiler) can be extended via plugins.  A plugin is
+// just a program that reads a CodeGeneratorRequest from stdin and writes a
+// CodeGeneratorResponse to stdout.
+//
+// Plugins written using C++ can use google/protobuf/compiler/plugin.h instead
+// of dealing with the raw protocol defined here.
+//
+// A plugin executable needs only to be placed somewhere in the path.  The
+// plugin should be named "protoc-gen-$NAME", and will then be used when the
+// flag "--${NAME}_out" is passed to protoc.
+
+package google.protobuf.compiler;
+
+import "google/protobuf/descriptor.proto";
+
+// An encoded CodeGeneratorRequest is written to the plugin's stdin.
+message CodeGeneratorRequest {
+  // The .proto files that were explicitly listed on the command-line.  The
+  // code generator should generate code only for these files.  Each file's
+  // descriptor will be included in proto_file, below.
+  repeated string file_to_generate = 1;
+
+  // The generator parameter passed on the command-line.
+  optional string parameter = 2;
+
+  // FileDescriptorProtos for all files in files_to_generate and everything
+  // they import.  The files will appear in topological order, so each file
+  // appears before any file that imports it.
+  //
+  // protoc guarantees that all proto_files will be written after
+  // the fields above, even though this is not technically guaranteed by the
+  // protobuf wire format.  This theoretically could allow a plugin to stream
+  // in the FileDescriptorProtos and handle them one by one rather than read
+  // the entire set into memory at once.  However, as of this writing, this
+  // is not similarly optimized on protoc's end -- it will store all fields in
+  // memory at once before sending them to the plugin.
+  repeated FileDescriptorProto proto_file = 15;
+}
+
+// The plugin writes an encoded CodeGeneratorResponse to stdout.
+message CodeGeneratorResponse {
+  // Error message.  If non-empty, code generation failed.  The plugin process
+  // should exit with status code zero even if it reports an error in this way.
+  //
+  // This should be used to indicate errors in .proto files which prevent the
+  // code generator from generating correct code.  Errors which indicate a
+  // problem in protoc itself -- such as the input CodeGeneratorRequest being
+  // unparseable -- should be reported by writing a message to stderr and
+  // exiting with a non-zero status code.
+  optional string error = 1;
+
+  // Represents a single generated file.
+  message File {
+    // The file name, relative to the output directory.  The name must not
+    // contain "." or ".." components and must be relative, not be absolute (so,
+    // the file cannot lie outside the output directory).  "/" must be used as
+    // the path separator, not "\".
+    //
+    // If the name is omitted, the content will be appended to the previous
+    // file.  This allows the generator to break large files into small chunks,
+    // and allows the generated text to be streamed back to protoc so that large
+    // files need not reside completely in memory at one time.  Note that as of
+    // this writing protoc does not optimize for this -- it will read the entire
+    // CodeGeneratorResponse before writing files to disk.
+    optional string name = 1;
+
+    // If non-empty, indicates that the named file should already exist, and the
+    // content here is to be inserted into that file at a defined insertion
+    // point.  This feature allows a code generator to extend the output
+    // produced by another code generator.  The original generator may provide
+    // insertion points by placing special annotations in the file that look
+    // like:
+    //   @@protoc_insertion_point(NAME)
+    // The annotation can have arbitrary text before and after it on the line,
+    // which allows it to be placed in a comment.  NAME should be replaced with
+    // an identifier naming the point -- this is what other generators will use
+    // as the insertion_point.  Code inserted at this point will be placed
+    // immediately above the line containing the insertion point (thus multiple
+    // insertions to the same point will come out in the order they were added).
+    // The double-@ is intended to make it unlikely that the generated code
+    // could contain things that look like insertion points by accident.
+    //
+    // For example, the C++ code generator places the following line in the
+    // .pb.h files that it generates:
+    //   // @@protoc_insertion_point(namespace_scope)
+    // This line appears within the scope of the file's package namespace, but
+    // outside of any particular class.  Another plugin can then specify the
+    // insertion_point "namespace_scope" to generate additional classes or
+    // other declarations that should be placed in this scope.
+    //
+    // Note that if the line containing the insertion point begins with
+    // whitespace, the same whitespace will be added to every line of the
+    // inserted text.  This is useful for languages like Python, where
+    // indentation matters.  In these languages, the insertion point comment
+    // should be indented the same amount as any inserted code will need to be
+    // in order to work correctly in that context.
+    //
+    // The code generator that generates the initial file and the one which
+    // inserts into it must both run as part of a single invocation of protoc.
+    // Code generators are executed in the order in which they appear on the
+    // command line.
+    //
+    // If |insertion_point| is present, |name| must also be present.
+    optional string insertion_point = 2;
+
+    // The file contents.
+    optional string content = 15;
+  }
+  repeated File file = 15;
+}

generator/plugin_pb2.py

@@ -0,0 +1,161 @@
+# Generated by the protocol buffer compiler.  DO NOT EDIT!
+
+from google.protobuf import descriptor
+from google.protobuf import message
+from google.protobuf import reflection
+from google.protobuf import descriptor_pb2
+# @@protoc_insertion_point(imports)
+
+
+import google.protobuf.descriptor_pb2
+
+DESCRIPTOR = descriptor.FileDescriptor(
+  name='plugin.proto',
+  package='google.protobuf.compiler',
+  serialized_pb='\n\x0cplugin.proto\x12\x18google.protobuf.compiler\x1a google/protobuf/descriptor.proto\"}\n\x14\x43odeGeneratorRequest\x12\x18\n\x10\x66ile_to_generate\x18\x01 \x03(\t\x12\x11\n\tparameter\x18\x02 \x01(\t\x12\x38\n\nproto_file\x18\x0f \x03(\x0b\x32$.google.protobuf.FileDescriptorProto\"\xaa\x01\n\x15\x43odeGeneratorResponse\x12\r\n\x05\x65rror\x18\x01 \x01(\t\x12\x42\n\x04\x66ile\x18\x0f \x03(\x0b\x32\x34.google.protobuf.compiler.CodeGeneratorResponse.File\x1a>\n\x04\x46ile\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x17\n\x0finsertion_point\x18\x02 \x01(\t\x12\x0f\n\x07\x63ontent\x18\x0f \x01(\t')
+
+
+
+
+_CODEGENERATORREQUEST = descriptor.Descriptor(
+  name='CodeGeneratorRequest',
+  full_name='google.protobuf.compiler.CodeGeneratorRequest',
+  filename=None,
+  file=DESCRIPTOR,
+  containing_type=None,
+  fields=[
+    descriptor.FieldDescriptor(
+      name='file_to_generate', full_name='google.protobuf.compiler.CodeGeneratorRequest.file_to_generate', index=0,
+      number=1, type=9, cpp_type=9, label=3,
+      has_default_value=False, default_value=[],
+      message_type=None, enum_type=None, containing_type=None,
+      is_extension=False, extension_scope=None,
+      options=None),
+    descriptor.FieldDescriptor(
+      name='parameter', full_name='google.protobuf.compiler.CodeGeneratorRequest.parameter', index=1,
+      number=2, type=9, cpp_type=9, label=1,
+      has_default_value=False, default_value=unicode("", "utf-8"),
+      message_type=None, enum_type=None, containing_type=None,
+      is_extension=False, extension_scope=None,
+      options=None),
+    descriptor.FieldDescriptor(
+      name='proto_file', full_name='google.protobuf.compiler.CodeGeneratorRequest.proto_file', index=2,
+      number=15, type=11, cpp_type=10, label=3,
+      has_default_value=False, default_value=[],
+      message_type=None, enum_type=None, containing_type=None,
+      is_extension=False, extension_scope=None,
+      options=None),
+  ],
+  extensions=[
+  ],
+  nested_types=[],
+  enum_types=[
+  ],
+  options=None,
+  is_extendable=False,
+  extension_ranges=[],
+  serialized_start=76,
+  serialized_end=201,
+)
+
+
+_CODEGENERATORRESPONSE_FILE = descriptor.Descriptor(
+  name='File',
+  full_name='google.protobuf.compiler.CodeGeneratorResponse.File',
+  filename=None,
+  file=DESCRIPTOR,
+  containing_type=None,
+  fields=[
+    descriptor.FieldDescriptor(
+      name='name', full_name='google.protobuf.compiler.CodeGeneratorResponse.File.name', index=0,
+      number=1, type=9, cpp_type=9, label=1,
+      has_default_value=False, default_value=unicode("", "utf-8"),
+      message_type=None, enum_type=None, containing_type=None,
+      is_extension=False, extension_scope=None,
+      options=None),
+    descriptor.FieldDescriptor(
+      name='insertion_point', full_name='google.protobuf.compiler.CodeGeneratorResponse.File.insertion_point', index=1,
+      number=2, type=9, cpp_type=9, label=1,
+      has_default_value=False, default_value=unicode("", "utf-8"),
+      message_type=None, enum_type=None, containing_type=None,
+      is_extension=False, extension_scope=None,
+      options=None),
+    descriptor.FieldDescriptor(
+      name='content', full_name='google.protobuf.compiler.CodeGeneratorResponse.File.content', index=2,
+      number=15, type=9, cpp_type=9, label=1,
+      has_default_value=False, default_value=unicode("", "utf-8"),
+      message_type=None, enum_type=None, containing_type=None,
+      is_extension=False, extension_scope=None,
+      options=None),
+  ],
+  extensions=[
+  ],
+  nested_types=[],
+  enum_types=[
+  ],
+  options=None,
+  is_extendable=False,
+  extension_ranges=[],
+  serialized_start=312,
+  serialized_end=374,
+)
+
+_CODEGENERATORRESPONSE = descriptor.Descriptor(
+  name='CodeGeneratorResponse',
+  full_name='google.protobuf.compiler.CodeGeneratorResponse',
+  filename=None,
+  file=DESCRIPTOR,
+  containing_type=None,
+  fields=[
+    descriptor.FieldDescriptor(
+      name='error', full_name='google.protobuf.compiler.CodeGeneratorResponse.error', index=0,
+      number=1, type=9, cpp_type=9, label=1,
+      has_default_value=False, default_value=unicode("", "utf-8"),
+      message_type=None, enum_type=None, containing_type=None,
+      is_extension=False, extension_scope=None,
+      options=None),
+    descriptor.FieldDescriptor(
+      name='file', full_name='google.protobuf.compiler.CodeGeneratorResponse.file', index=1,
+      number=15, type=11, cpp_type=10, label=3,
+      has_default_value=False, default_value=[],
+      message_type=None, enum_type=None, containing_type=None,
+      is_extension=False, extension_scope=None,
+      options=None),
+  ],
+  extensions=[
+  ],
+  nested_types=[_CODEGENERATORRESPONSE_FILE, ],
+  enum_types=[
+  ],
+  options=None,
+  is_extendable=False,
+  extension_ranges=[],
+  serialized_start=204,
+  serialized_end=374,
+)
+
+_CODEGENERATORREQUEST.fields_by_name['proto_file'].message_type = google.protobuf.descriptor_pb2._FILEDESCRIPTORPROTO
+_CODEGENERATORRESPONSE_FILE.containing_type = _CODEGENERATORRESPONSE;
+_CODEGENERATORRESPONSE.fields_by_name['file'].message_type = _CODEGENERATORRESPONSE_FILE
+DESCRIPTOR.message_types_by_name['CodeGeneratorRequest'] = _CODEGENERATORREQUEST
+DESCRIPTOR.message_types_by_name['CodeGeneratorResponse'] = _CODEGENERATORRESPONSE
+
+class CodeGeneratorRequest(message.Message):
+  __metaclass__ = reflection.GeneratedProtocolMessageType
+  DESCRIPTOR = _CODEGENERATORREQUEST
+  
+  # @@protoc_insertion_point(class_scope:google.protobuf.compiler.CodeGeneratorRequest)
+
+class CodeGeneratorResponse(message.Message):
+  __metaclass__ = reflection.GeneratedProtocolMessageType
+  
+  class File(message.Message):
+    __metaclass__ = reflection.GeneratedProtocolMessageType
+    DESCRIPTOR = _CODEGENERATORRESPONSE_FILE
+    
+    # @@protoc_insertion_point(class_scope:google.protobuf.compiler.CodeGeneratorResponse.File)
+  DESCRIPTOR = _CODEGENERATORRESPONSE
+  
+  # @@protoc_insertion_point(class_scope:google.protobuf.compiler.CodeGeneratorResponse)
+
+# @@protoc_insertion_point(module_scope)

pb.h

@@ -43,7 +43,7 @@
 
 /* Version of the nanopb library. Just in case you want to check it in
  * your own program. */
-#define NANOPB_VERSION nanopb-0.2.3-dev
+#define NANOPB_VERSION nanopb-0.2.4
 
 /* Include all the system headers needed by nanopb. You will need the
  * definitions of the following:
@@ -173,6 +173,20 @@ typedef uint8_t pb_type_t;
 #define PB_HTYPE(x) ((x) & PB_HTYPE_MASK)
 #define PB_LTYPE(x) ((x) & PB_LTYPE_MASK)
 
+/* Data type used for storing sizes of struct fields
+ * and array counts.
+ */
+#if defined(PB_FIELD_32BIT)
+    typedef uint32_t pb_size_t;
+    typedef int32_t pb_ssize_t;
+#elif defined(PB_FIELD_16BIT)
+    typedef uint16_t pb_size_t;
+    typedef int16_t pb_ssize_t;
+#else
+    typedef uint8_t pb_size_t;
+    typedef int8_t pb_ssize_t;
+#endif
+
 /* This structure is used in auto-generated constants
  * to specify struct fields.
  * You can change field sizes if you need structures
@@ -184,29 +198,12 @@ typedef uint8_t pb_type_t;
 PB_PACKED_STRUCT_START
 typedef struct _pb_field_t pb_field_t;
 struct _pb_field_t {
-
+    pb_size_t tag;
-#if !defined(PB_FIELD_16BIT) && !defined(PB_FIELD_32BIT)
-    uint8_t tag;
     pb_type_t type;
-    uint8_t data_offset; /* Offset of field data, relative to previous field. */
+    pb_size_t data_offset; /* Offset of field data, relative to previous field. */
-    int8_t size_offset; /* Offset of array size or has-boolean, relative to data */
+    pb_ssize_t size_offset; /* Offset of array size or has-boolean, relative to data */
-    uint8_t data_size; /* Data size in bytes for a single item */
+    pb_size_t data_size; /* Data size in bytes for a single item */
-    uint8_t array_size; /* Maximum number of entries in array */
+    pb_size_t array_size; /* Maximum number of entries in array */
-#elif defined(PB_FIELD_16BIT) && !defined(PB_FIELD_32BIT)
-    uint16_t tag;
-    pb_type_t type;
-    uint8_t data_offset;
-    int8_t size_offset;
-    uint16_t data_size;
-    uint16_t array_size;
-#else
-    uint32_t tag;
-    pb_type_t type;
-    uint8_t data_offset;
-    int8_t size_offset;
-    uint32_t data_size;
-    uint32_t array_size;
-#endif
     
     /* Field definitions for submessage
      * OR default value for all other non-array, non-callback types
@@ -331,58 +328,69 @@ struct _pb_extension_t {
 };
 
 /* These macros are used to declare pb_field_t's in the constant array. */
+/* Size of a structure member, in bytes. */
 #define pb_membersize(st, m) (sizeof ((st*)0)->m)
+/* Number of entries in an array. */
 #define pb_arraysize(st, m) (pb_membersize(st, m) / pb_membersize(st, m[0]))
+/* Delta from start of one member to the start of another member. */
 #define pb_delta(st, m1, m2) ((int)offsetof(st, m1) - (int)offsetof(st, m2))
-#define pb_delta_end(st, m1, m2) (int)(offsetof(st, m1) == offsetof(st, m2) \
+/* Marks the end of the field list */
-                                  ? offsetof(st, m1) \
-                                  : offsetof(st, m1) - offsetof(st, m2) - pb_membersize(st, m2))
 #define PB_LAST_FIELD {0,(pb_type_t) 0,0,0,0,0,0}
 
+/* Macros for filling in the data_offset field */
+/* data_offset for first field in a message */
+#define PB_DATAOFFSET_FIRST(st, m1, m2) (offsetof(st, m1))
+/* data_offset for subsequent fields */
+#define PB_DATAOFFSET_OTHER(st, m1, m2) (offsetof(st, m1) - offsetof(st, m2) - pb_membersize(st, m2))
+/* Choose first/other based on m1 == m2 (deprecated, remains for backwards compatibility) */
+#define PB_DATAOFFSET_CHOOSE(st, m1, m2) (int)(offsetof(st, m1) == offsetof(st, m2) \
+                                  ? PB_DATAOFFSET_FIRST(st, m1, m2) \
+                                  : PB_DATAOFFSET_OTHER(st, m1, m2))
+
 /* Required fields are the simplest. They just have delta (padding) from
  * previous field end, and the size of the field. Pointer is used for
  * submessages and default values.
  */
-#define PB_REQUIRED_STATIC(tag, st, m, pm, ltype, ptr) \
+#define PB_REQUIRED_STATIC(tag, st, m, fd, ltype, ptr) \
     {tag, PB_ATYPE_STATIC | PB_HTYPE_REQUIRED | ltype, \
-    pb_delta_end(st, m, pm), 0, pb_membersize(st, m), 0, ptr}
+    fd, 0, pb_membersize(st, m), 0, ptr}
 
 /* Optional fields add the delta to the has_ variable. */
-#define PB_OPTIONAL_STATIC(tag, st, m, pm, ltype, ptr) \
+#define PB_OPTIONAL_STATIC(tag, st, m, fd, ltype, ptr) \
     {tag, PB_ATYPE_STATIC | PB_HTYPE_OPTIONAL | ltype, \
-    pb_delta_end(st, m, pm), \
+    fd, \
     pb_delta(st, has_ ## m, m), \
     pb_membersize(st, m), 0, ptr}
 
 /* Repeated fields have a _count field and also the maximum number of entries. */
-#define PB_REPEATED_STATIC(tag, st, m, pm, ltype, ptr) \
+#define PB_REPEATED_STATIC(tag, st, m, fd, ltype, ptr) \
     {tag, PB_ATYPE_STATIC | PB_HTYPE_REPEATED | ltype, \
-    pb_delta_end(st, m, pm), \
+    fd, \
     pb_delta(st, m ## _count, m), \
     pb_membersize(st, m[0]), \
     pb_arraysize(st, m), ptr}
 
 /* Callbacks are much like required fields except with special datatype. */
-#define PB_REQUIRED_CALLBACK(tag, st, m, pm, ltype, ptr) \
+#define PB_REQUIRED_CALLBACK(tag, st, m, fd, ltype, ptr) \
     {tag, PB_ATYPE_CALLBACK | PB_HTYPE_REQUIRED | ltype, \
-    pb_delta_end(st, m, pm), 0, pb_membersize(st, m), 0, ptr}
+    fd, 0, pb_membersize(st, m), 0, ptr}
 
-#define PB_OPTIONAL_CALLBACK(tag, st, m, pm, ltype, ptr) \
+#define PB_OPTIONAL_CALLBACK(tag, st, m, fd, ltype, ptr) \
     {tag, PB_ATYPE_CALLBACK | PB_HTYPE_OPTIONAL | ltype, \
-    pb_delta_end(st, m, pm), 0, pb_membersize(st, m), 0, ptr}
+    fd, 0, pb_membersize(st, m), 0, ptr}
     
-#define PB_REPEATED_CALLBACK(tag, st, m, pm, ltype, ptr) \
+#define PB_REPEATED_CALLBACK(tag, st, m, fd, ltype, ptr) \
     {tag, PB_ATYPE_CALLBACK | PB_HTYPE_REPEATED | ltype, \
-    pb_delta_end(st, m, pm), 0, pb_membersize(st, m), 0, ptr}
+    fd, 0, pb_membersize(st, m), 0, ptr}
 
 /* Optional extensions don't have the has_ field, as that would be redundant. */
-#define PB_OPTEXT_STATIC(tag, st, m, pm, ltype, ptr) \
+#define PB_OPTEXT_STATIC(tag, st, m, fd, ltype, ptr) \
     {tag, PB_ATYPE_STATIC | PB_HTYPE_OPTIONAL | ltype, \
     0, \
     0, \
     pb_membersize(st, m), 0, ptr}
 
-#define PB_OPTEXT_CALLBACK(tag, st, m, pm, ltype, ptr) \
+#define PB_OPTEXT_CALLBACK(tag, st, m, fd, ltype, ptr) \
     {tag, PB_ATYPE_CALLBACK | PB_HTYPE_OPTIONAL | ltype, \
     0, 0, pb_membersize(st, m), 0, ptr}
 
@@ -421,8 +429,21 @@ struct _pb_extension_t {
  */
 
 #define PB_FIELD(tag, type, rules, allocation, message, field, prevfield, ptr) \
-    PB_ ## rules ## _ ## allocation(tag, message, field, prevfield, \
+    PB_ ## rules ## _ ## allocation(tag, message, field, \
-                                    PB_LTYPE_MAP_ ## type, ptr)
+        PB_DATAOFFSET_CHOOSE(message, field, prevfield), \
+        PB_LTYPE_MAP_ ## type, ptr)
+
+/* This is a new version of the macro used by nanopb generator from
+ * version 0.2.3 onwards. It avoids the use of a ternary expression in
+ * the initialization, which confused some compilers.
+ *
+ * - Placement: FIRST or OTHER, depending on if this is the first field in structure.
+ *
+ */
+#define PB_FIELD2(tag, type, rules, allocation, placement, message, field, prevfield, ptr) \
+    PB_ ## rules ## _ ## allocation(tag, message, field, \
+        PB_DATAOFFSET_ ## placement(message, field, prevfield), \
+        PB_LTYPE_MAP_ ## type, ptr)
 
 
 /* These macros are used for giving out error messages.

pb_decode.c

@@ -3,11 +3,13 @@
  * 2011 Petteri Aimonen <jpa@kapsi.fi>
  */
 
-/* The warn_unused_result attribute appeared first in gcc-3.4.0 */
+/* Use the GCC warn_unused_result attribute to check that all return values
+ * are propagated correctly. On other compilers and gcc before 3.4.0 just
+ * ignore the annotation.
+ */
 #if !defined(__GNUC__) || ( __GNUC__ < 3) || (__GNUC__ == 3 && __GNUC_MINOR__ < 4)
     #define checkreturn
 #else
-    /* Verify that we remember to check all return values for proper error propagation */
     #define checkreturn __attribute__((warn_unused_result))
 #endif
 
@@ -15,8 +17,46 @@
 #include "pb.h"
 #include "pb_decode.h"
 
+/**************************************
+ * Declarations internal to this file *
+ **************************************/
+
+/* Iterator for pb_field_t list */
+typedef struct {
+    const pb_field_t *start; /* Start of the pb_field_t array */
+    const pb_field_t *pos; /* Current position of the iterator */
+    unsigned field_index; /* Zero-based index of the field. */
+    unsigned required_field_index; /* Zero-based index that counts only the required fields */
+    void *dest_struct; /* Pointer to the destination structure to decode to */
+    void *pData; /* Pointer where to store current field value */
+    void *pSize; /* Pointer where to store the size of current array field */
+} pb_field_iterator_t;
+
 typedef bool (*pb_decoder_t)(pb_istream_t *stream, const pb_field_t *field, void *dest) checkreturn;
 
+static bool checkreturn buf_read(pb_istream_t *stream, uint8_t *buf, size_t count);
+static bool checkreturn pb_decode_varint32(pb_istream_t *stream, uint32_t *dest);
+static bool checkreturn read_raw_value(pb_istream_t *stream, pb_wire_type_t wire_type, uint8_t *buf, size_t *size);
+static void pb_field_init(pb_field_iterator_t *iter, const pb_field_t *fields, void *dest_struct);
+static bool pb_field_next(pb_field_iterator_t *iter);
+static bool checkreturn pb_field_find(pb_field_iterator_t *iter, uint32_t tag);
+static bool checkreturn decode_static_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iterator_t *iter);
+static bool checkreturn decode_callback_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iterator_t *iter);
+static bool checkreturn decode_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iterator_t *iter);
+static bool checkreturn default_extension_decoder(pb_istream_t *stream, pb_extension_t *extension, uint32_t tag, pb_wire_type_t wire_type);
+static bool checkreturn decode_extension(pb_istream_t *stream, uint32_t tag, pb_wire_type_t wire_type, pb_field_iterator_t *iter);
+static bool checkreturn find_extension_field(pb_field_iterator_t *iter);
+static void pb_message_set_to_defaults(const pb_field_t fields[], void *dest_struct);
+static bool pb_dec_varint(pb_istream_t *stream, const pb_field_t *field, void *dest);
+static bool checkreturn pb_dec_svarint(pb_istream_t *stream, const pb_field_t *field, void *dest);
+static bool checkreturn pb_dec_fixed32(pb_istream_t *stream, const pb_field_t *field, void *dest);
+static bool checkreturn pb_dec_fixed64(pb_istream_t *stream, const pb_field_t *field, void *dest);
+static bool checkreturn pb_dec_bytes(pb_istream_t *stream, const pb_field_t *field, void *dest);
+static bool checkreturn pb_dec_string(pb_istream_t *stream, const pb_field_t *field, void *dest);
+static bool checkreturn pb_dec_submessage(pb_istream_t *stream, const pb_field_t *field, void *dest);
+static bool checkreturn pb_skip_varint(pb_istream_t *stream);
+static bool checkreturn pb_skip_string(pb_istream_t *stream);
+
 /* --- Function pointers to field decoders ---
  * Order in the array must match pb_action_t LTYPE numbering.
  */
@@ -32,9 +72,9 @@ static const pb_decoder_t PB_DECODERS[PB_LTYPES_COUNT] = {
     NULL /* extensions */
 };
 
-/**************
+/*******************************
- * pb_istream *
+ * pb_istream_t implementation *
- **************/
+ *******************************/
 
 static bool checkreturn buf_read(pb_istream_t *stream, uint8_t *buf, size_t count)
 {
@@ -277,17 +317,6 @@ void pb_close_string_substream(pb_istream_t *stream, pb_istream_t *substream)
 #endif
 }
 
-/* Iterator for pb_field_t list */
-typedef struct {
-    const pb_field_t *start; /* Start of the pb_field_t array */
-    const pb_field_t *pos; /* Current position of the iterator */
-    unsigned field_index; /* Zero-based index of the field. */
-    unsigned required_field_index; /* Zero-based index that counts only the required fields */
-    void *dest_struct; /* Pointer to the destination structure to decode to */
-    void *pData; /* Pointer where to store current field value */
-    void *pSize; /* Pointer where to store the size of current array field */
-} pb_field_iterator_t;
-
 static void pb_field_init(pb_field_iterator_t *iter, const pb_field_t *fields, void *dest_struct)
 {
     iter->start = iter->pos = fields;
@@ -478,7 +507,7 @@ static bool checkreturn decode_field(pb_istream_t *stream, pb_wire_type_t wire_t
 
 /* Default handler for extension fields. Expects a pb_field_t structure
  * in extension->type->arg. */
-static bool checkreturn default_extension_handler(pb_istream_t *stream,
+static bool checkreturn default_extension_decoder(pb_istream_t *stream,
     pb_extension_t *extension, uint32_t tag, pb_wire_type_t wire_type)
 {
     const pb_field_t *field = (const pb_field_t*)extension->type->arg;
@@ -513,7 +542,7 @@ static bool checkreturn decode_extension(pb_istream_t *stream,
         if (extension->type->decode)
             status = extension->type->decode(stream, extension, tag, wire_type);
         else
-            status = default_extension_handler(stream, extension, tag, wire_type);
+            status = default_extension_decoder(stream, extension, tag, wire_type);
 
         if (!status)
             return false;

pb_decode.h

@@ -131,25 +131,6 @@ bool pb_decode_fixed64(pb_istream_t *stream, void *dest);
 bool pb_make_string_substream(pb_istream_t *stream, pb_istream_t *substream);
 void pb_close_string_substream(pb_istream_t *stream, pb_istream_t *substream);
 
-
-/*******************************
- * Internal / legacy functions *
- *******************************/
-
-#ifdef NANOPB_INTERNALS
-bool pb_dec_varint(pb_istream_t *stream, const pb_field_t *field, void *dest);
-bool pb_dec_svarint(pb_istream_t *stream, const pb_field_t *field, void *dest);
-bool pb_dec_fixed32(pb_istream_t *stream, const pb_field_t *field, void *dest);
-bool pb_dec_fixed64(pb_istream_t *stream, const pb_field_t *field, void *dest);
-
-bool pb_dec_bytes(pb_istream_t *stream, const pb_field_t *field, void *dest);
-bool pb_dec_string(pb_istream_t *stream, const pb_field_t *field, void *dest);
-bool pb_dec_submessage(pb_istream_t *stream, const pb_field_t *field, void *dest);
-
-bool pb_skip_varint(pb_istream_t *stream);
-bool pb_skip_string(pb_istream_t *stream);
-#endif
-
 #ifdef __cplusplus
 } /* extern "C" */
 #endif

pb_encode.c

@@ -7,16 +7,34 @@
 #include "pb.h"
 #include "pb_encode.h"
 
-/* The warn_unused_result attribute appeared first in gcc-3.4.0 */
+/* Use the GCC warn_unused_result attribute to check that all return values
+ * are propagated correctly. On other compilers and gcc before 3.4.0 just
+ * ignore the annotation.
+ */
 #if !defined(__GNUC__) || ( __GNUC__ < 3) || (__GNUC__ == 3 && __GNUC_MINOR__ < 4)
     #define checkreturn
 #else
-    /* Verify that we remember to check all return values for proper error propagation */
     #define checkreturn __attribute__((warn_unused_result))
 #endif
 
+/**************************************
+ * Declarations internal to this file *
+ **************************************/
 typedef bool (*pb_encoder_t)(pb_ostream_t *stream, const pb_field_t *field, const void *src) checkreturn;
 
+static bool checkreturn buf_write(pb_ostream_t *stream, const uint8_t *buf, size_t count);
+static bool checkreturn encode_array(pb_ostream_t *stream, const pb_field_t *field, const void *pData, size_t count, pb_encoder_t func);
+static bool checkreturn encode_field(pb_ostream_t *stream, const pb_field_t *field, const void *pData);
+static bool checkreturn default_extension_encoder(pb_ostream_t *stream, const pb_extension_t *extension);
+static bool checkreturn encode_extension_field(pb_ostream_t *stream, const pb_field_t *field, const void *pData);
+static bool checkreturn pb_enc_varint(pb_ostream_t *stream, const pb_field_t *field, const void *src);
+static bool checkreturn pb_enc_svarint(pb_ostream_t *stream, const pb_field_t *field, const void *src);
+static bool checkreturn pb_enc_fixed32(pb_ostream_t *stream, const pb_field_t *field, const void *src);
+static bool checkreturn pb_enc_fixed64(pb_ostream_t *stream, const pb_field_t *field, const void *src);
+static bool checkreturn pb_enc_bytes(pb_ostream_t *stream, const pb_field_t *field, const void *src);
+static bool checkreturn pb_enc_string(pb_ostream_t *stream, const pb_field_t *field, const void *src);
+static bool checkreturn pb_enc_submessage(pb_ostream_t *stream, const pb_field_t *field, const void *src);
+
 /* --- Function pointers to field encoders ---
  * Order in the array must match pb_action_t LTYPE numbering.
  */
@@ -32,7 +50,9 @@ static const pb_encoder_t PB_ENCODERS[PB_LTYPES_COUNT] = {
     NULL /* extensions */
 };
 
-/* pb_ostream_t implementation */
+/*******************************
+ * pb_ostream_t implementation *
+ *******************************/
 
 static bool checkreturn buf_write(pb_ostream_t *stream, const uint8_t *buf, size_t count)
 {
@@ -49,7 +69,7 @@ pb_ostream_t pb_ostream_from_buffer(uint8_t *buf, size_t bufsize)
 {
     pb_ostream_t stream;
 #ifdef PB_BUFFER_ONLY
-    stream.callback = (void*)1; /* Just some marker value */
+    stream.callback = (void*)1; /* Just a marker value */
 #else
     stream.callback = &buf_write;
 #endif
@@ -82,7 +102,9 @@ bool checkreturn pb_write(pb_ostream_t *stream, const uint8_t *buf, size_t count
     return true;
 }
 
-/* Main encoding stuff */
+/*************************
+ * Encode a single field *
+ *************************/
 
 /* Encode a static array. Handles the size calculations and possible packing. */
 static bool checkreturn encode_array(pb_ostream_t *stream, const pb_field_t *field,
@@ -95,6 +117,9 @@ static bool checkreturn encode_array(pb_ostream_t *stream, const pb_field_t *fie
     if (count == 0)
         return true;
         
+    if (count > field->array_size)
+        PB_RETURN_ERROR(stream, "array max size exceeded");
+    
     /* We always pack arrays if the datatype allows it. */
     if (PB_LTYPE(field->type) <= PB_LTYPE_LAST_PACKABLE)
     {
@@ -242,7 +267,7 @@ static bool checkreturn encode_field(pb_ostream_t *stream,
 
 /* Default handler for extension fields. Expects to have a pb_field_t
  * pointer in the extension->type->arg field. */
-static bool checkreturn default_extension_handler(pb_ostream_t *stream,
+static bool checkreturn default_extension_encoder(pb_ostream_t *stream,
     const pb_extension_t *extension)
 {
     const pb_field_t *field = (const pb_field_t*)extension->type->arg;
@@ -263,7 +288,7 @@ static bool checkreturn encode_extension_field(pb_ostream_t *stream,
         if (extension->type->encode)
             status = extension->type->encode(stream, extension);
         else
-            status = default_extension_handler(stream, extension);
+            status = default_extension_encoder(stream, extension);
 
         if (!status)
             return false;
@@ -274,6 +299,10 @@ static bool checkreturn encode_extension_field(pb_ostream_t *stream,
     return true;
 }
 
+/*********************
+ * Encode all fields *
+ *********************/
+
 bool checkreturn pb_encode(pb_ostream_t *stream, const pb_field_t fields[], const void *src_struct)
 {
     const pb_field_t *field = fields;
@@ -316,7 +345,9 @@ bool pb_encode_delimited(pb_ostream_t *stream, const pb_field_t fields[], const
     return pb_encode_submessage(stream, fields, src_struct);
 }
 
-/* Helper functions */
+/********************
+ * Helper functions *
+ ********************/
 bool checkreturn pb_encode_varint(pb_ostream_t *stream, uint64_t value)
 {
     uint8_t buffer[10];
@@ -518,7 +549,10 @@ bool checkreturn pb_enc_fixed32(pb_ostream_t *stream, const pb_field_t *field, c
 bool checkreturn pb_enc_bytes(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
     const pb_bytes_array_t *bytes = (const pb_bytes_array_t*)src;
-    UNUSED(field);
+
+    if (bytes->size + offsetof(pb_bytes_array_t, bytes) > field->data_size)
+        PB_RETURN_ERROR(stream, "bytes size exceeded");
+    
     return pb_encode_string(stream, bytes->bytes, bytes->size);
 }
 

pb_encode.h

@@ -143,24 +143,6 @@ bool pb_encode_fixed64(pb_ostream_t *stream, const void *value);
  */
 bool pb_encode_submessage(pb_ostream_t *stream, const pb_field_t fields[], const void *src_struct);
 
-
-/*******************************
- * Internal / legacy functions *
- *******************************/
-
-#ifdef NANOPB_INTERNALS
-bool pb_enc_varint(pb_ostream_t *stream, const pb_field_t *field, const void *src);
-bool pb_enc_svarint(pb_ostream_t *stream, const pb_field_t *field, const void *src);
-bool pb_enc_fixed32(pb_ostream_t *stream, const pb_field_t *field, const void *src);
-bool pb_enc_fixed64(pb_ostream_t *stream, const pb_field_t *field, const void *src);
-bool pb_enc_bytes(pb_ostream_t *stream, const pb_field_t *field, const void *src);
-bool pb_enc_string(pb_ostream_t *stream, const pb_field_t *field, const void *src);
-#endif
-
-/* This function is not recommended for new programs. Use pb_encode_submessage()
- * instead, it has the same functionality with a less confusing interface. */
-bool pb_enc_submessage(pb_ostream_t *stream, const pb_field_t *field, const void *src);
-
 #ifdef __cplusplus
 } /* extern "C" */
 #endif

tests/SConstruct

@@ -73,11 +73,16 @@ if 'gcc' in env['CC']:
     env.Append(CFLAGS = '-ansi -pedantic -g -O0 -Wall -Werror --coverage -fstack-protector-all')
     env.Append(LINKFLAGS = '--coverage')
     
+    # We currently need uint64_t anyway, even though ANSI C90 otherwise..
+    env.Append(CFLAGS = '-Wno-long-long')
+
     # More strict checks on the nanopb core
     env.Append(CORECFLAGS = '-Wextra -Wcast-qual -Wlogical-op -Wconversion')
+    env.Append(CORECFLAGS = ' -Wmissing-prototypes -Wmissing-declarations -Wredundant-decls')
+    env.Append(CORECFLAGS = ' -Wstack-protector')
 elif 'clang' in env['CC']:
     # CLang
-    env.Append(CFLAGS = '-ansi -pedantic -g -O0 -Wall -Werror')
+    env.Append(CFLAGS = '-ansi -g -O0 -Wall -Werror')
     env.Append(CORECFLAGS = ' -Wextra -Wcast-qual -Wconversion')
 elif 'cl' in env['CC']:
     # Microsoft Visual C++

tests/alltypes/encode_alltypes.c

@@ -115,7 +115,7 @@ int main(int argc, char **argv)
     alltypes.end = 1099;
     
     {
-        uint8_t buffer[1024];
+        uint8_t buffer[AllTypes_size];
         pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
         
         /* Now encode it and check if we succeeded. */

tests/basic_buffer/decode_buffer.c

@@ -60,7 +60,7 @@ bool print_person(pb_istream_t *stream)
 
 int main()
 {
-    uint8_t buffer[512];
+    uint8_t buffer[Person_size];
     pb_istream_t stream;
     size_t count;
     

tests/basic_buffer/encode_buffer.c

@@ -10,7 +10,7 @@
 
 int main()
 {
-    uint8_t buffer[512];
+    uint8_t buffer[Person_size];
     pb_ostream_t stream;
     
     /* Initialize the structure with constants */

tests/common/unittestproto.proto

@@ -12,6 +12,10 @@ message StringMessage {
     required string data = 1 [(nanopb).max_size = 10];
 }
 
+message BytesMessage {
+    required bytes data = 1 [(nanopb).max_size = 16];
+}
+
 message CallbackArray {
     // We cheat a bit and use this message for testing other types, too.
     // Nanopb does not care about the actual defined data type for callback

tests/decode_unittests/SConscript

@@ -1,4 +1,4 @@
 Import('env')
-p = env.Program(["decode_unittests.c", "#common/unittestproto.pb.c", "#common/pb_decode.o"])
+p = env.Program(["decode_unittests.c", "#common/unittestproto.pb.c"])
 env.RunTest(p)
 

tests/decode_unittests/decode_unittests.c

@@ -1,8 +1,8 @@
-#define NANOPB_INTERNALS
+/* This includes the whole .c file to get access to static functions. */
+#include "pb_decode.c"
 
 #include <stdio.h>
 #include <string.h>
-#include "pb_decode.h"
 #include "unittests.h"
 #include "unittestproto.pb.h"
 

tests/encode_unittests/SConscript

@@ -1,5 +1,5 @@
 # Build and run the stand-alone unit tests for the nanopb encoder part.
 
 Import('env')
-p = env.Program(["encode_unittests.c", "#common/unittestproto.pb.c", "#common/pb_encode.o"])
+p = env.Program(["encode_unittests.c", "#common/unittestproto.pb.c"])
 env.RunTest(p)

tests/encode_unittests/encode_unittests.c

@@ -1,8 +1,8 @@
-#define NANOPB_INTERNALS
+/* This includes the whole .c file to get access to static functions. */
+#include "pb_encode.c"
 
 #include <stdio.h>
 #include <string.h>
-#include "pb_encode.h"
 #include "unittests.h"
 #include "unittestproto.pb.h"
 
@@ -172,9 +172,9 @@ int main()
         struct { size_t size; uint8_t bytes[5]; } value = {5, {'x', 'y', 'z', 'z', 'y'}};
     
         COMMENT("Test pb_enc_bytes")
-        TEST(WRITES(pb_enc_bytes(&s, NULL, &value), "\x05xyzzy"))
+        TEST(WRITES(pb_enc_bytes(&s, &BytesMessage_fields[0], &value), "\x05xyzzy"))
         value.size = 0;
-        TEST(WRITES(pb_enc_bytes(&s, NULL, &value), "\x00"))
+        TEST(WRITES(pb_enc_bytes(&s, &BytesMessage_fields[0], &value), "\x00"))
     }
     
     {
@@ -223,6 +223,20 @@ int main()
         TEST(!pb_encode(&s, FloatArray_fields, &msg))
     }
     
+    {
+        uint8_t buffer[50];
+        pb_ostream_t s;
+        FloatArray msg = {1, {99.0f}};
+        
+        COMMENT("Test array size limit in pb_encode")
+        
+        s = pb_ostream_from_buffer(buffer, sizeof(buffer));
+        TEST((msg.data_count = 10) && pb_encode(&s, FloatArray_fields, &msg))
+        
+        s = pb_ostream_from_buffer(buffer, sizeof(buffer));
+        TEST((msg.data_count = 11) && !pb_encode(&s, FloatArray_fields, &msg))
+    }
+    
     {
         uint8_t buffer[10];
         pb_ostream_t s;
@@ -244,6 +258,20 @@ int main()
                     "\x0A\x07\x0A\x05\x01\x02\x03\x04\x05"))
     }
     
+    {
+        uint8_t buffer[32];
+        pb_ostream_t s;
+        BytesMessage msg = {{3, "xyz"}};
+        
+        COMMENT("Test pb_encode with bytes message.")
+        TEST(WRITES(pb_encode(&s, BytesMessage_fields, &msg),
+                    "\x0A\x03xyz"))
+        
+        msg.data.size = 17; /* More than maximum */
+        TEST(!pb_encode(&s, BytesMessage_fields, &msg))
+    }
+        
+    
     {
         uint8_t buffer[20];
         pb_ostream_t s;
@@ -280,6 +308,19 @@ int main()
         TEST(!pb_encode(&s, CallbackContainerContainer_fields, &msg2))
     }
     
+    {
+        uint8_t buffer[StringMessage_size];
+        pb_ostream_t s;
+        StringMessage msg = {"0123456789"};
+        
+        s = pb_ostream_from_buffer(buffer, sizeof(buffer));
+        
+        COMMENT("Test that StringMessage_size is correct")
+
+        TEST(pb_encode(&s, StringMessage_fields, &msg));
+        TEST(s.bytes_written == StringMessage_size);
+    }
+    
     if (status != 0)
         fprintf(stdout, "\n\nSome tests FAILED!\n");
     

tests/extensions/encode_extensions.c

@@ -46,5 +46,9 @@ int main(int argc, char **argv)
         fprintf(stderr, "Encoding failed: %s\n", PB_GET_ERROR(&stream));
         return 1; /* Failure */
     }
+    
+    /* Check that the field tags are properly generated */
+    (void)AllTypes_extensionfield1_tag;
+    (void)ExtensionMessage_AllTypes_extensionfield2_tag;
 }
 

tests/extra_fields/SConscript

@@ -6,5 +6,9 @@ dec = env.GetBuildPath('#basic_buffer/${PROGPREFIX}decode_buffer${PROGSUFFIX}')
 env.RunTest('person_with_extra_field.output', [dec, "person_with_extra_field.pb"])
 env.Compare(["person_with_extra_field.output", "person_with_extra_field.expected"])
 
+dec = env.GetBuildPath('#basic_stream/${PROGPREFIX}decode_stream${PROGSUFFIX}')
+env.RunTest('person_with_extra_field_stream.output', [dec, "person_with_extra_field.pb"])
+env.Compare(["person_with_extra_field_stream.output", "person_with_extra_field.expected"])
+
 dec2 = env.GetBuildPath('#alltypes/${PROGPREFIX}decode_alltypes${PROGSUFFIX}')
 env.RunTest('alltypes_with_extra_fields.output', [dec2, 'alltypes_with_extra_fields.pb'])

tests/message_sizes/SConscript

@@ -0,0 +1,11 @@
+# Test the generation of message size #defines
+
+Import('env')
+
+incpath = env.Clone()
+incpath.Append(PROTOCPATH = '#message_sizes')
+
+incpath.NanopbProto("messages1")
+incpath.NanopbProto("messages2")
+
+incpath.Program(['dummy.c', 'messages1.pb.c', 'messages2.pb.c'])

tests/message_sizes/dummy.c

@@ -0,0 +1,9 @@
+/* Just test that the file can be compiled successfully. */
+
+#include "messages2.pb.h"
+
+int main()
+{
+    return xmit_size;
+}
+

tests/message_sizes/messages1.proto

@@ -0,0 +1,27 @@
+enum MessageStatus {
+    FAIL = 0;
+    OK = 1;
+};
+
+message MessageInfo {
+   required fixed32 msg_id = 1;
+   optional fixed32 interface_id = 2;
+}
+
+message MessageResponseInfo {
+   required fixed64 interface_id = 1;
+   required fixed32 seq = 2;
+   required fixed32 msg_id = 3;
+}
+
+message MessageHeader {
+   required MessageInfo info = 1;
+   optional MessageResponseInfo response_info = 2;
+   optional MessageResponse response = 3;
+}
+
+message MessageResponse {
+   required MessageStatus status = 1;
+   required fixed32 seq = 2;
+}
+

tests/message_sizes/messages2.proto

@@ -0,0 +1,8 @@
+import 'nanopb.proto';
+import 'messages1.proto';
+
+message xmit {
+   required MessageHeader header = 1;
+   required bytes data = 2 [(nanopb).max_size = 128];
+}
+

tests/site_scons/site_init.py

@@ -37,7 +37,8 @@ def add_nanopb_builders(env):
                                   src_suffix = '.pb',
                                   emitter = nanopb_targets)
     env.Append(BUILDERS = {'Nanopb': nanopb_file_builder})
-    env.SetDefault(NANOPB_GENERATOR = 'python ' + env.GetBuildPath("#../generator/nanopb_generator.py"))
+    gen_path = env['ESCAPE'](env.GetBuildPath("#../generator/nanopb_generator.py"))
+    env.SetDefault(NANOPB_GENERATOR = 'python ' + gen_path)
     env.SetDefault(NANOPB_FLAGS = '-q')
 
     # Combined method to run both protoc and nanopb generator
@@ -71,8 +72,10 @@ def add_nanopb_builders(env):
 
     # Build command that decodes a message using protoc
     def decode_actions(source, target, env, for_signature):
-        dirs = ' '.join(['-I' + env.GetBuildPath(d) for d in env['PROTOCPATH']])
+        esc = env['ESCAPE']
-        return '$PROTOC $PROTOCFLAGS %s --decode=%s %s <%s >%s' % (dirs, env['MESSAGE'], source[1], source[0], target[0])
+        dirs = ' '.join(['-I' + esc(env.GetBuildPath(d)) for d in env['PROTOCPATH']])
+        return '$PROTOC $PROTOCFLAGS %s --decode=%s %s <%s >%s' % (
+            dirs, env['MESSAGE'], esc(str(source[1])), esc(str(source[0])), esc(str(target[0])))
 
     decode_builder = Builder(generator = decode_actions,
                              suffix = '.decoded')