Publishing nanopb-0.3.1

Possible consequences of bug:
1) Denial of service by causing a crash
   Possible when all of the following apply:
      - Untrusted data is passed to pb_decode()
      - The top-level message contains a static string field as the first field.
   Causes a single write of '0' byte to 1 byte before the message struct.

2) Remote code execution
   Possible when all of the following apply:
      - 64-bit platform
      - The message or a submessage contains a static/pointer string field.
      - Decoding directly from a custom pb_istream_t
      - bytes_left on the stream is set to larger than 4 GB
   Causes a write of up to 4 GB of data past the string field.

3) Possible heap corruption or remote code execution
   Possible when all of the following apply:
      - less than 64-bit platform
      - The message or a submessage contains a pointer-type bytes field.
   Causes a write of sizeof(pb_size_t) bytes of data past a 0-byte long
   malloc()ed buffer. On many malloc() implementations, this causes at
   most a crash. However, remote code execution through a controlled jump
   cannot be ruled out.

--

Detailed analysis follows

In the following consideration, I define "platform bitness" as equal to
number of bits in size_t datatype. Therefore most 8-bit platforms are
regarded as 16-bit for the purposes of this discussion.

1. The overflow in pb_dec_string

The overflow happens in this computation:

uint32_t size;
size_t alloc_size;
alloc_size = size + 1;

There are two ways in which the overflow can occur: In the uint32_t
addition, or in the cast to size_t. This depends on the platform
bitness.

On 32- and 64-bit platforms, the size has to be UINT32_MAX for the
overflow to occur. In that case alloc_size will be 0.

On 16-bit platforms, overflow will happen whenever size is more than
UINT16_MAX, and resulting alloc_size is attacker controlled.

For static fields, the alloc_size value is just checked against the
field data size. For pointer fields, the alloc_size value is passed to
malloc(). End result in both cases is the same, the storage is 0 or
just a few bytes in length.

On 16-bit platforms, another overflow occurs in the call to pb_read(),
when passing the original size. An attacker will want the passed value
to be larger than the alloc_size, therefore the only reasonable choice
is to have size = UINT16_MAX and alloc_size = 0. Any larger multiple
will truncate to the same values.

At this point we have read atleast the tag and the string length of the
message, i.e. atleast 3 bytes. The maximum initial value for stream
bytes_left is SIZE_MAX, thus at this point at most SIZE_MAX-3 bytes are
remaining.

On 32-bit and 16-bit platforms this means that the size passed to
pb_read() is always larger than the number of remaining bytes. This
causes pb_read() to fail immediately, before reading any bytes.

On 64-bit platforms, it is possible for the bytes_left value to be set
to a value larger than UINT32_MAX, which is the wraparound point in
size calculation. In this case pb_read() will succeed and write up to 4
GB of attacker controlled data over the RAM that comes after the string
field.

On all platforms, there is an unconditional write of a terminating null
byte. Because the size of size_t typically reflects the size of the
processor address space, a write at UINT16_MAX or UINT32_MAX bytes
after the string field actually wraps back to before the string field.
Consequently, on 32-bit and 16-bit platforms, the bug causes a single
write of '0' byte at one byte before the string field.

If the string field is in the middle of a message, this will just
corrupt other data in the message struct. Because the message contents
is attacker controlled anyway, this is a non-issue. However, if the
string field is the first field in the top-level message, it can
corrupt other data on the stack/heap before it. Typically a single '0'
write at a location not controlled by attacker is enough only for a
denial-of-service attack.

When using pointer fields and malloc(), the attacker controlled
alloc_size will cause a 0-size allocation to happen. By the same logic
as before, on 32-bit and 16-bit platforms this causes a '0' byte write
only. On 64-bit platforms, however, it will again allow up to 4 GB of
malicious data to be written over memory, if the stream length allows
the read.

2. The overflow in pb_dec_bytes

This overflow happens in the PB_BYTES_ARRAY_T_ALLOCSIZE macro:

The computation is done in size_t data type this time. This means that
an overflow is possible only when n is larger than SIZE_MAX -
offsetof(..). The offsetof value in this case is equal to
sizeof(pb_size_t) bytes.

Because the incoming size value is limited to 32 bits, no overflow can
happen here on 64-bit platforms.

The size will be passed to pb_read(). Like before, on 32-bit and 16-bit
platforms the read will always fail before writing anything.

This leaves only the write of bdest->size as exploitable. On statically
allocated fields, the size field will always be allocated, regardless
of alloc_size. In this case, no buffer overflow is possible here, but
user code could possibly use the attacker controlled size value and
read past a buffer.

If the field is allocated through malloc(), this will allow a write of
sizeof(pb_size_t) attacker controlled bytes to past a 0-byte long
buffer. In typical malloc implementations, this will either fit in
unused alignment padding area, or cause a heap corruption and a crash.
Under very exceptional situation it could allow attacker to influence
the behaviour of malloc(), possibly jumping into an attacker-controlled
location and thus leading to remote code execution.

.gitignore

@@ -5,35 +5,24 @@
 *.pb.c
 *.pb.h
 *.pb
+*.pyc
+*_pb2.py
 *~
 *.tar.gz
+.sconsign.dblite
+config.log
+.sconf_temp
+tests/build
 julkaisu.txt
+dist
 docs/*.html
 docs/generator_flow.png
-example/client
-example/server
-example_avr_double/decode_double
-example_avr_double/encode_double
-example_avr_double/test_conversions
-example_unions/decode
-example_unions/encode
+examples/simple/simple
+examples/network_server/client
+examples/network_server/server
+examples/using_double_on_avr/decode_double
+examples/using_double_on_avr/encode_double
+examples/using_double_on_avr/test_conversions
+examples/using_union_messages/decode
+examples/using_union_messages/encode
 generator/nanopb_pb2.pyc
-tests/decode_unittests
-tests/encode_unittests
-tests/test_compiles
-tests/test_decode1
-tests/test_decode2
-tests/test_decode3
-tests/test_decode3_buf
-tests/test_decode_callbacks
-tests/test_encode1
-tests/test_encode2
-tests/test_encode3
-tests/test_encode3_buf
-tests/test_encode_callbacks
-tests/test_missing_fields
-tests/test_multiple_files
-tests/bc_decode
-tests/bc_encode
-tests/breakpoints
-

CHANGELOG.txt

@@ -1,4 +1,84 @@
-nanopb-0.2.2
+nanopb-0.3.1 (2014-09-11)
+ Fix security issue due to size_t overflows. (issue 132)
+ Fix memory leak with duplicated fields and PB_ENABLE_MALLOC
+ Fix crash if pb_release() is called twice.
+ Fix cyclic message support (issue 130)
+ Fix error in generated initializers for repeated pointer fields.
+ Improve tests (issues 113, 126)
+
+nanopb-0.3.0 (2014-08-26)
+ NOTE: See docs/migration.html or online at
+ http://koti.kapsi.fi/~jpa/nanopb/docs/migration.html
+ for changes in this version. Most importantly, you need to add
+ pb_common.c to the list of files to compile.
+
+ Separated field iterator logic to pb_common.c (issue 128)
+ Change the _count fields to use pb_size_t datatype (issue 82)
+ Added PB_ prefix to macro names (issue 106)
+ Added #if version guard to generated files (issue 129)
+ Added migration document
+
+nanopb-0.2.9 (2014-08-09)
+ NOTE: If you are using the -e option with the generator, you have
+ to prepend . to the argument to get the same behaviour as before.
+
+ Do not automatically add a dot with generator -e option. (issue 122)
+ Fix problem with .options file and extension fields. (issue 125)
+ Don't use SIZE_MAX macro, as it is not in C89. (issue 120)
+ Generate #defines for initializing message structures. (issue 79)
+ Add skip_message option to generator. (issue 121)
+ Add PB_PACKED_STRUCT support for Keil MDK-ARM toolchain (issue 119)
+ Give better messages about the .options file path. (issue 124)
+ Improved tests
+
+nanopb-0.2.8 (2014-05-20)
+ Fix security issue with PB_ENABLE_MALLOC. (issue 117)
+ Add option to not add timestamps to .pb.h and .pb.c preambles. (issue 115)
+ Documentation updates
+ Improved tests
+
+nanopb-0.2.7 (2014-04-07)
+ Fix bug with default values for extension fields (issue 111)
+ Fix some MISRA-C warnings (issue 91)
+ Implemented optional malloc() support (issue 80)
+ Changed pointer-type bytes field datatype
+ Add a "found" field to pb_extension_t (issue 112)
+ Add convenience function pb_get_encoded_size() (issue 16)
+
+nanopb-0.2.6 (2014-02-15)
+ Fix generator error with bytes callback fields (issue 99)
+ Fix warnings about large integer constants (issue 102)
+ Add comments to where STATIC_ASSERT is used (issue 96)
+ Add warning about unknown field names on .options (issue 105)
+ Move descriptor.proto to google/protobuf subdirectory (issue 104)
+ Improved tests
+
+nanopb-0.2.5 (2014-01-01)
+ Fix a bug with encoding negative values in int32 fields (issue 97)
+ Create binary packages of the generator + dependencies (issue 47)
+ Add support for pointer-type fields to the encoder (part of issue 80)
+ Fixed path in FindNanopb.cmake (issue 94)
+ Improved tests
+
+nanopb-0.2.4 (2013-11-07)
+ Remove the deprecated NANOPB_INTERNALS functions from public API.
+ Document the security model.
+ Check array and bytes max sizes when encoding (issue 90)
+ Add #defines for maximum encoded message size (issue 89)
+ Add #define tags for extension fields (issue 93)
+ Fix MISRA C violations (issue 91)
+ Clean up pb_field_t definition with typedefs.
+
+nanopb-0.2.3 (2013-09-18)
+ Improve compatibility by removing ternary operator from initializations (issue 88)
+ Fix build error on Visual C++ (issue 84, patch by Markus Schwarzenberg)
+ Don't stop on unsupported extension fields (issue 83)
+ Add an example pb_syshdr.h file for non-C99 compilers
+ Reorganize tests and examples into subfolders (issue 63)
+ Switch from Makefiles to scons for building the tests
+ Make the tests buildable on Windows
+
+nanopb-0.2.2 (2013-08-18)
  Add support for extension fields (issue 17)
  Fix unknown fields in empty message (issue 78)
  Include the field tags in the generated .pb.h file.
@@ -7,7 +87,7 @@ nanopb-0.2.2
  Documentation improvements (issues 12, 77 and others)
  Improved tests
 
-nanopb-0.2.1
+nanopb-0.2.1 (2013-04-14)
  NOTE: The default callback function signature has changed.
        If you don't want to update your code, define PB_OLD_CALLBACK_STYLE.
  
@@ -24,7 +104,7 @@ nanopb-0.2.1
  Various new generator options
  Improved tests
 
-nanopb-0.2.0
+nanopb-0.2.0 (2013-03-02)
  NOTE: This release requires you to regenerate all .pb.c
        files. Files generated by older versions will not
        compile anymore.
@@ -37,7 +117,7 @@ nanopb-0.2.0
  Add option to give file extension to generator (by Michael Haberler)
  Documentation updates
 
-nanopb-0.1.9
+nanopb-0.1.9 (2013-02-13)
  Fixed error message bugs (issues 52, 56)
  Sanitize #ifndef filename (issue 50)
  Performance improvements
@@ -48,13 +128,13 @@ nanopb-0.1.9
  Added generator option to make message structs packed. (issue 49)
  Add more test cases.
 
-nanopb-0.1.8
+nanopb-0.1.8 (2012-12-13)
  Fix bugs in the enum short names introduced in 0.1.7 (issues 42, 43)
  Fix STATIC_ASSERT macro when using multiple .proto files. (issue 41)
  Fix missing initialization of istream.errmsg
  Make tests/Makefile work for non-gcc compilers (issue 40)
 
-nanopb-0.1.7
+nanopb-0.1.7 (2012-11-11)
  Remove "skip" mode from pb_istream_t callbacks. Example implementation had a bug. (issue 37)
  Add option to use shorter names for enum values (issue 38)
  Improve options support in generator (issues 12, 30)
@@ -64,19 +144,19 @@ nanopb-0.1.7
  Add buffer size check in example (issue 34)
  Fix build warnings on MS compilers (issue 33)
 
-nanopb-0.1.6
+nanopb-0.1.6 (2012-09-02)
  Reorganize the field decoder interface (issue 2)
  Improve performance in submessage decoding (issue 28)
  Implement error messages in the decoder side (issue 7)
  Extended testcases (alltypes test is now complete).
  Fix some compiler warnings (issues 25, 26, 27, 32).
 
-nanopb-0.1.5
+nanopb-0.1.5 (2012-08-04)
  Fix bug in decoder with packed arrays (issue 23).
  Extended testcases.
  Fix some compiler warnings.
 
-nanopb-0.1.4
+nanopb-0.1.4 (2012-07-05)
  Add compile-time options for easy-to-use >255 field support.
  Improve the detection of missing required fields.
  Added example on how to handle union messages.
@@ -84,20 +164,20 @@ nanopb-0.1.4
  Fix problems that stopped the code from compiling with some compilers.
  Fix some compiler warnings.
 
-nanopb-0.1.3
+nanopb-0.1.3 (2012-06-12)
  Refactor the field encoder interface.
  Improve generator error messages (issue 5)
  Add descriptor.proto into the #include exclusion list
  Fix some compiler warnings.
 
-nanopb-0.1.2
+nanopb-0.1.2 (2012-02-15)
  Make the generator to generate include for other .proto files (issue 4).
  Fixed generator not working on Windows (issue 3)
 
-nanopb-0.1.1
+nanopb-0.1.1 (2012-01-14)
  Fixed bug in encoder with 'bytes' fields (issue 1).
  Fixed a bug in the generator that caused a compiler error on sfixed32 and sfixed64 fields.
  Extended testcases.
 
-nanopb-0.1.0
+nanopb-0.1.0 (2012-01-06)
  First stable release.

README

@@ -1,11 +0,0 @@
-Nanopb is a small code-size Protocol Buffers implementation.
-
-Homepage: http://kapsi.fi/~jpa/nanopb/
-
-To compile the library, you'll need these libraries:
-protobuf-compiler python-protobuf libprotobuf-dev
-
-The only runtime dependencies are memset() and memcpy().
-
-To run the tests, run make under the tests folder.
-If it completes without error, everything is fine.

README.txt

@@ -0,0 +1,61 @@
+Nanopb is a small code-size Protocol Buffers implementation in ansi C. It is
+especially suitable for use in microcontrollers, but fits any memory
+restricted system.
+
+Homepage: http://kapsi.fi/~jpa/nanopb/
+
+
+
+
+Using the nanopb library
+========================
+To use the nanopb library, you need to do two things:
+
+1) Compile your .proto files for nanopb, using protoc.
+2) Include pb_encode.c and pb_decode.c in your project.
+
+The easiest way to get started is to study the project in "examples/simple".
+It contains a Makefile, which should work directly under most Linux systems.
+However, for any other kind of build system, see the manual steps in
+README.txt in that folder.
+
+
+
+Using the Protocol Buffers compiler (protoc)
+============================================
+The nanopb generator is implemented as a plugin for the Google's own protoc
+compiler. This has the advantage that there is no need to reimplement the
+basic parsing of .proto files. However, it does mean that you need the
+Google's protobuf library in order to run the generator.
+
+If you have downloaded a binary package for nanopb (either Windows, Linux or
+Mac OS X version), the 'protoc' binary is included in the 'generator-bin'
+folder. In this case, you are ready to go. Simply run this command:
+
+    generator-bin/protoc --nanopb_out=. myprotocol.proto
+
+However, if you are using a git checkout or a plain source distribution, you
+need to provide your own version of protoc and the Google's protobuf library.
+On Linux, the necessary packages are protobuf-compiler and python-protobuf.
+On Windows, you can either build Google's protobuf library from source or use
+one of the binary distributions of it. In either case, if you use a separate
+protoc, you need to manually give the path to nanopb generator:
+
+    protoc --plugin=protoc-gen-nanopb=nanopb/generator/protoc-gen-nanopb ...
+
+
+
+Running the tests
+=================
+If you want to perform further development of the nanopb core, or to verify
+its functionality using your compiler and platform, you'll want to run the
+test suite. The build rules for the test suite are implemented using Scons,
+so you need to have that installed. To run the tests:
+
+    cd tests
+    scons
+
+This will show the progress of various test cases. If the output does not
+end in an error, the test cases were successful.
+
+

docs/Makefile

@@ -1,4 +1,4 @@
-all: index.html concepts.html reference.html \
+all: index.html concepts.html reference.html security.html migration.html \
 	generator_flow.png
 
 %.png: %.svg

docs/index.rst

@@ -47,7 +47,6 @@ Features and limitations
 
 **Limitations**
 
-#) User must provide callbacks when decoding arrays or strings without maximum size. Malloc support could be added as a separate module.
 #) Some speed has been sacrificed for code size.
 #) Encoding is focused on writing to streams. For memory buffers only it could be made more efficient.
 #) The deprecated Protocol Buffers feature called "groups" is not supported.
@@ -90,22 +89,38 @@ After that, buffer will contain the encoded message.
 The number of bytes in the message is stored in *stream.bytes_written*.
 You can feed the message to *protoc --decode=Example message.proto* to verify its validity.
 
-For complete examples of the simple cases, see *tests/test_decode1.c* and *tests/test_encode1.c*. For an example with network interface, see the *example* subdirectory.
+For a complete example of the simple case, see *example/simple.c*.
+For a more complex example with network interface, see the *example/network_server* subdirectory.
 
 Compiler requirements
 =====================
-Nanopb should compile with most ansi-C compatible compilers. It however requires a few header files to be available:
+Nanopb should compile with most ansi-C compatible compilers. It however
+requires a few header files to be available:
 
 #) *string.h*, with these functions: *strlen*, *memcpy*, *memset*
 #) *stdint.h*, for definitions of *int32_t* etc.
 #) *stddef.h*, for definition of *size_t*
 #) *stdbool.h*, for definition of *bool*
 
-If these header files do not come with your compiler, you should be able to find suitable replacements online. Mostly the requirements are very simple, just a few basic functions and typedefs.
+If these header files do not come with your compiler, you can use the
+file *extra/pb_syshdr.h* instead. It contains an example of how to provide
+the dependencies. You may have to edit it a bit to suit your custom platform.
 
-Alternatively, you can define *PB_SYSTEM_HEADER*, which should be the name of a single header file including all the necessary definitions.
+To use the pb_syshdr.h, define *PB_SYSTEM_HEADER* as *"pb_syshdr.h"* (including the quotes).
+Similarly, you can provide a custom include file, which should provide all the dependencies
+listed above.
 
-Debugging and testing
-=====================
-Extensive unittests are included under the *tests* folder. Just type *make* there to run the tests.
+Running the test cases
+======================
+Extensive unittests and test cases are included under the *tests* folder.
+
+To build the tests, you will need the `scons`__ build system. The tests should
+be runnable on most platforms. Windows and Linux builds are regularly tested.
+
+__ http://www.scons.org/
+
+In addition to the build system, you will also need a working Google Protocol
+Buffers *protoc* compiler, and the Python bindings for Protocol Buffers. On
+Debian-based systems, install the following packages: *protobuf-compiler*,
+*python-protobuf* and *libprotobuf-dev*.
 

docs/menu.rst

@@ -3,8 +3,11 @@
     1) `Overview`_
     2) `Concepts`_
     3) `API reference`_
+    4) `Security model`_
+    5) `Migration from older versions`_
     
 .. _`Overview`: index.html
 .. _`Concepts`: concepts.html
 .. _`API reference`: reference.html
-
+.. _`Security model`: security.html
+.. _`Migration from older versions`: migration.html

docs/migration.rst

@@ -0,0 +1,239 @@
+=====================================
+Nanopb: Migration from older versions
+=====================================
+
+.. include :: menu.rst
+
+This document details all the breaking changes that have been made to nanopb
+since its initial release. For each change, the rationale and required
+modifications of user applications are explained. Also any error indications
+are included, in order to make it easier to find this document.
+
+.. contents ::
+
+Nanopb-0.3.0 (2014-08-26)
+=========================
+
+Separate field iterator logic to pb_common.c
+--------------------------------------------
+**Rationale:** Originally, the field iteration logic was simple enough to be
+duplicated in *pb_decode.c* and *pb_encode.c*. New field types have made the
+logic more complex, which required the creation of a new file to contain the
+common functionality.
+
+**Changes:** There is a new file, *pb_common.c*, which must be included in
+builds.
+
+**Required actions:** Add *pb_common.c* to build rules. This file is always
+required. Either *pb_decode.c* or *pb_encode.c* can still be left out if some
+functionality is not needed.
+
+**Error indications:** Linker error: undefined reference to
+*pb_field_iter_begin*, *pb_field_iter_next* or similar.
+
+Change data type of field counts to pb_size_t
+---------------------------------------------
+**Rationale:** Often nanopb is used with small arrays, such as 255 items or
+less. Using a full *size_t* field to store the array count wastes memory if
+there are many arrays. There already exists parameters *PB_FIELD_16BIT* and
+*PB_FIELD_32BIT* which tell nanopb what is the maximum size of arrays in use.
+
+**Changes:** Generator will now use *pb_size_t* for the array *_count* fields.
+The size of the type will be controlled by the *PB_FIELD_16BIT* and
+*PB_FIELD_32BIT* compilation time options.
+
+**Required actions:** Regenerate all *.pb.h* files. In some cases casts to the
+*pb_size_t* type may need to be added in the user code when accessing the
+*_count* fields.
+
+**Error indications:** Incorrect data at runtime, crashes. But note that other
+changes in the same version already require regenerating the files and have
+better indications of errors, so this is only an issue for development
+versions.
+
+Renamed some macros and identifiers
+-----------------------------------
+**Rationale:** Some names in nanopb core were badly chosen and conflicted with
+ISO C99 reserved names or lacked a prefix. While they haven't caused trouble
+so far, it is reasonable to switch to non-conflicting names as these are rarely
+used from user code.
+
+**Changes:** The following identifier names have changed:
+
+  * Macros:
+  
+    * STATIC_ASSERT(x) -> PB_STATIC_ASSERT(x)
+    * UNUSED(x) -> PB_UNUSED(x)
+  
+  * Include guards:
+  
+    * _PB_filename_ -> PB_filename_INCLUDED
+  
+  * Structure forward declaration tags:
+  
+    * _pb_field_t -> pb_field_s
+    * _pb_bytes_array_t -> pb_bytes_array_s
+    * _pb_callback_t -> pb_callback_s
+    * _pb_extension_type_t -> pb_extension_type_s
+    * _pb_extension_t -> pb_extension_s
+    * _pb_istream_t -> pb_istream_s
+    * _pb_ostream_t -> pb_ostream_s
+
+**Required actions:** Regenerate all *.pb.c* files. If you use any of the above
+identifiers in your application code, perform search-replace to the new name.
+
+**Error indications:** Compiler errors on lines with the macro/type names.
+
+Nanopb-0.2.9 (2014-08-09)
+=========================
+
+Change semantics of generator -e option
+---------------------------------------
+**Rationale:** Some compilers do not accept filenames with two dots (like
+in default extension .pb.c). The *-e* option to the generator allowed changing
+the extension, but not skipping the extra dot.
+
+**Changes:** The *-e* option in generator will no longer add the prepending
+dot. The default value has been adjusted accordingly to *.pb.c* to keep the
+default behaviour the same as before.
+
+**Required actions:** Only if using the generator -e option. Add dot before
+the parameter value on the command line.
+
+**Error indications:** File not found when trying to compile generated files.
+
+Nanopb-0.2.7 (2014-04-07)
+=========================
+
+Changed pointer-type bytes field datatype
+-----------------------------------------
+**Rationale:** In the initial pointer encoding support since nanopb-0.2.5,
+the bytes type used a separate *pb_bytes_ptr_t* type to represent *bytes*
+fields. This made it easy to encode data from a separate, user-allocated
+buffer. However, it made the internal logic more complex and was inconsistent
+with the other types.
+
+**Changes:** Dynamically allocated bytes fields now have the *pb_bytes_array_t*
+type, just like statically allocated ones.
+
+**Required actions:** Only if using pointer-type fields with the bytes datatype.
+Change any access to *msg->field.size* to *msg->field->size*. Change any
+allocation to reserve space of amount *PB_BYTES_ARRAY_T_ALLOCSIZE(n)*. If the
+data pointer was begin assigned from external source, implement the field using
+a callback function instead.
+
+**Error indications:** Compiler error: unknown type name *pb_bytes_ptr_t*.
+
+Nanopb-0.2.4 (2013-11-07)
+=========================
+
+Remove the NANOPB_INTERNALS compilation option
+----------------------------------------------
+**Rationale:** Having the option in the headers required the functions to
+be non-static, even if the option is not used. This caused errors on some
+static analysis tools.
+
+**Changes:** The *#ifdef* and associated functions were removed from the
+header.
+
+**Required actions:** Only if the *NANOPB_INTERNALS* option was previously
+used. Actions are as listed under nanopb-0.1.3 and nanopb-0.1.6.
+
+**Error indications:** Compiler warning: implicit declaration of function
+*pb_dec_string*, *pb_enc_string*, or similar.
+
+Nanopb-0.2.1 (2013-04-14)
+=========================
+
+Callback function signature
+---------------------------
+**Rationale:** Previously the auxilary data to field callbacks was passed
+as *void\**. This allowed passing of any data, but made it unnecessarily
+complex to return a pointer from callback.
+
+**Changes:** The callback function parameter was changed to *void\*\**.
+
+**Required actions:** You can continue using the old callback style by
+defining *PB_OLD_CALLBACK_STYLE*. Recommended action is to:
+
+  * Change the callback signatures to contain *void\*\** for decoders and
+    *void \* const \** for encoders.
+  * Change the callback function body to use *\*arg* instead of *arg*.
+
+**Error indications:** Compiler warning: assignment from incompatible
+pointer type, when initializing *funcs.encode* or *funcs.decode*.
+
+Nanopb-0.2.0 (2013-03-02)
+=========================
+
+Reformatted generated .pb.c file using macros
+---------------------------------------------
+**Rationale:** Previously the generator made a list of C *pb_field_t*
+initializers in the .pb.c file. This led to a need to regenerate all .pb.c
+files after even small changes to the *pb_field_t* definition.
+
+**Changes:** Macros were added to pb.h which allow for cleaner definition
+of the .pb.c contents. By changing the macro definitions, changes to the
+field structure are possible without breaking compatibility with old .pb.c
+files.
+
+**Required actions:** Regenerate all .pb.c files from the .proto sources.
+
+**Error indications:** Compiler warning: implicit declaration of function
+*pb_delta_end*.
+
+Changed pb_type_t definitions
+-----------------------------
+**Rationale:** The *pb_type_t* was previously an enumeration type. This
+caused warnings on some compilers when using bitwise operations to set flags
+inside the values.
+
+**Changes:** The *pb_type_t* was changed to *typedef uint8_t*. The values
+were changed to *#define*. Some value names were changed for consistency.
+
+**Required actions:** Only if you directly access the `pb_field_t` contents
+in your own code, something which is not usually done. Needed changes:
+
+  * Change *PB_HTYPE_ARRAY* to *PB_HTYPE_REPEATED*.
+  * Change *PB_HTYPE_CALLBACK* to *PB_ATYPE()* and *PB_ATYPE_CALLBACK*.
+
+**Error indications:** Compiler error: *PB_HTYPE_ARRAY* or *PB_HTYPE_CALLBACK*
+undeclared.
+
+Nanopb-0.1.6 (2012-09-02)
+=========================
+
+Refactored field decoder interface
+----------------------------------
+**Rationale:** Similarly to field encoders in nanopb-0.1.3.
+
+**Changes:** New functions with names *pb_decode_\** were added.
+
+**Required actions:** By defining NANOPB_INTERNALS, you can still keep using
+the old functions. Recommended action is to replace any calls with the newer
+*pb_decode_\** equivalents.
+
+**Error indications:** Compiler warning: implicit declaration of function
+*pb_dec_string*, *pb_dec_varint*, *pb_dec_submessage* or similar.
+
+Nanopb-0.1.3 (2012-06-12)
+=========================
+
+Refactored field encoder interface
+----------------------------------
+**Rationale:** The old *pb_enc_\** functions were designed mostly for the
+internal use by the core. Because they are internally accessed through
+function pointers, their signatures had to be common. This led to a confusing
+interface for external users.
+
+**Changes:** New functions with names *pb_encode_\** were added. These have
+easier to use interfaces. The old functions are now only thin wrappers for
+the new interface.
+
+**Required actions:** By defining NANOPB_INTERNALS, you can still keep using
+the old functions. Recommended action is to replace any calls with the newer
+*pb_encode_\** equivalents.
+
+**Error indications:** Compiler warning: implicit declaration of function
+*pb_enc_string*, *pb_enc_varint, *pb_enc_submessage* or similar.
+

docs/reference.rst

@@ -24,8 +24,8 @@ __BIG_ENDIAN__                 Set this if your platform stores integers and
                                floats in big-endian format. Mixed-endian
                                systems (different layout for ints and floats)
                                are currently not supported.
-NANOPB_INTERNALS               Set this to expose the field encoder functions
-                               that are hidden since nanopb-0.1.3.
+PB_ENABLE_MALLOC               Set this to enable dynamic allocation support
+                               in the decoder.
 PB_MAX_REQUIRED_FIELDS         Maximum number of required fields to check for
                                presence. Default value is 64. Increases stack
                                usage 1 byte per every 8 fields. Compiler
@@ -75,8 +75,9 @@ max_count                      Allocated number of entries in arrays
                                (*repeated* fields).
 type                           Type of the generated field. Default value
                                is *FT_DEFAULT*, which selects automatically.
-                               You can use *FT_CALLBACK*, *FT_STATIC* or
-                               *FT_IGNORE* to force a callback field, a static
+                               You can use *FT_CALLBACK*, *FT_POINTER*,
+                               *FT_STATIC* or *FT_IGNORE* to force a callback
+                               field, a dynamically allocated field, a static
                                field or to completely ignore the field.
 long_names                     Prefix the enum name to the enum value in
                                definitions, i.e. *EnumName_EnumValue*. Enabled
@@ -415,6 +416,17 @@ Encodes the contents of a structure as a protocol buffers message and writes it
 
 Normally pb_encode simply walks through the fields description array and serializes each field in turn. However, submessages must be serialized twice: first to calculate their size and then to actually write them to output. This causes some constraints for callback fields, which must return the same data on every call.
 
+pb_encode_delimited
+-------------------
+Calculates the length of the message, encodes it as varint and then encodes the message. ::
+
+    bool pb_encode_delimited(pb_ostream_t *stream, const pb_field_t fields[], const void *src_struct);
+
+(parameters are the same as for `pb_encode`_.)
+
+A common way to indicate the message length in Protocol Buffers is to prefix it with a varint.
+This function does this, and it is compatible with *parseDelimitedFrom* in Google's protobuf library.
+
 .. sidebar:: Encoding fields manually
 
     The functions with names *pb_encode_\** are used when dealing with callback fields. The typical reason for using callbacks is to have an array of unlimited size. In that case, `pb_encode`_ will call your callback function, which in turn will call *pb_encode_\** functions repeatedly to write out values.
@@ -577,6 +589,10 @@ In addition to EOF, the pb_decode implementation supports terminating a message
 
 For optional fields, this function applies the default value and sets *has_<field>* to false if the field is not present.
 
+If *PB_ENABLE_MALLOC* is defined, this function may allocate storage for any pointer type fields.
+In this case, you have to call `pb_release`_ to release the memory after you are done with the message.
+On error return `pb_decode` will release the memory itself.
+
 pb_decode_noinit
 ----------------
 Same as `pb_decode`_, except does not apply the default values to fields. ::
@@ -587,6 +603,35 @@ Same as `pb_decode`_, except does not apply the default values to fields. ::
 
 The destination structure should be filled with zeros before calling this function. Doing a *memset* manually can be slightly faster than using `pb_decode`_ if you don't need any default values.
 
+In addition to decoding a single message, this function can be used to merge two messages, so that
+values from previous message will remain if the new message does not contain a field.
+
+This function *will not* release the message even on error return. If you use *PB_ENABLE_MALLOC*,
+you will need to call `pb_release`_ yourself.
+
+pb_decode_delimited
+-------------------
+Same as `pb_decode`_, except that it first reads a varint with the length of the message. ::
+
+    bool pb_decode_delimited(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct);
+
+(parameters are the same as for `pb_decode`_.)
+
+A common method to indicate message size in Protocol Buffers is to prefix it with a varint.
+This function is compatible with *writeDelimitedTo* in the Google's Protocol Buffers library.
+
+pb_release
+----------
+Releases any dynamically allocated fields.
+
+    void pb_release(const pb_field_t fields[], void *dest_struct);
+
+:fields:        A field description array. Usually autogenerated.
+:dest_struct:   Pointer to structure where data will be stored.
+
+This function is only available if *PB_ENABLE_MALLOC* is defined. It will release any
+pointer type fields in the structure and set the pointers to NULL.
+
 pb_skip_varint
 --------------
 Skip a varint_ encoded integer without decoding it. ::

docs/security.rst

@@ -0,0 +1,81 @@
+======================
+Nanopb: Security model
+======================
+
+.. include :: menu.rst
+
+.. contents ::
+
+
+
+Importance of security in a Protocol Buffers library
+====================================================
+In the context of protocol buffers, security comes into play when decoding
+untrusted data. Naturally, if the attacker can modify the contents of a
+protocol buffers message, he can feed the application any values possible.
+Therefore the application itself must be prepared to receive untrusted values.
+
+Where nanopb plays a part is preventing the attacker from running arbitrary
+code on the target system. Mostly this means that there must not be any
+possibility to cause buffer overruns, memory corruption or invalid pointers
+by the means of crafting a malicious message.
+
+Division of trusted and untrusted data
+======================================
+The following data is regarded as **trusted**. It must be under the control of
+the application writer. Malicious data in these structures could cause
+security issues, such as execution of arbitrary code:
+
+1. Callback, pointer and extension fields in message structures given to
+   pb_encode() and pb_decode(). These fields are memory pointers, and are
+   generated depending on the message definition in the .proto file.
+2. The automatically generated field definitions, i.e. *pb_field_t* lists.
+3. Contents of the *pb_istream_t* and *pb_ostream_t* structures (this does not
+   mean the contents of the stream itself, just the stream definition).
+
+The following data is regarded as **untrusted**. Invalid/malicious data in
+these will cause "garbage in, garbage out" behaviour. It will not cause
+buffer overflows, information disclosure or other security problems:
+
+1. All data read from *pb_istream_t*.
+2. All fields in message structures, except callbacks, pointers and extensions.
+   (Beginning with nanopb-0.2.4, in earlier versions the field sizes are partially unchecked.)
+
+Invariants
+==========
+The following invariants are maintained during operation, even if the
+untrusted data has been maliciously crafted:
+
+1. Nanopb will never read more than *bytes_left* bytes from *pb_istream_t*.
+2. Nanopb will never write more than *max_size* bytes to *pb_ostream_t*.
+3. Nanopb will never access memory out of bounds of the message structure.
+4. After pb_decode() returns successfully, the message structure will be
+   internally consistent:
+
+   - The *count* fields of arrays will not exceed the array size.
+   - The *size* field of bytes will not exceed the allocated size.
+   - All string fields will have null terminator.
+
+5. After pb_encode() returns successfully, the resulting message is a valid
+   protocol buffers message. (Except if user-defined callbacks write incorrect
+   data.)
+
+Further considerations
+======================
+Even if the nanopb library is free of any security issues, there are still
+several possible attack vectors that the application author must consider.
+The following list is not comprehensive:
+
+1. Stack usage may depend on the contents of the message. The message
+   definition places an upper bound on how much stack will be used. Tests
+   should be run with all fields present, to record the maximum possible
+   stack usage.
+2. Callbacks can do anything. The code for the callbacks must be carefully
+   checked if they are used with untrusted data.
+3. If using stream input, a maximum size should be set in *pb_istream_t* to
+   stop a denial of service attack from using an infinite message.
+4. If using network sockets as streams, a timeout should be set to stop
+   denial of service attacks.
+5. If using *malloc()* support, some method of limiting memory use should be
+   employed. This can be done by defining custom *pb_realloc()* function.
+   Nanopb will properly detect and handle failed memory allocations.

example/Makefile

@@ -1,14 +0,0 @@
-CFLAGS=-ansi -Wall -Werror -I .. -g -O0
-DEPS=../pb_decode.c ../pb_decode.h ../pb_encode.c ../pb_encode.h ../pb.h
-
-all: server client
-
-clean:
-	rm -f server client fileproto.pb.c fileproto.pb.h
-
-%: %.c $(DEPS) fileproto.pb.h fileproto.pb.c
-	$(CC) $(CFLAGS) -o $@ $< ../pb_decode.c ../pb_encode.c fileproto.pb.c common.c
-
-fileproto.pb.c fileproto.pb.h: fileproto.proto ../generator/nanopb_generator.py
-	protoc -I. -I../generator -I/usr/include -ofileproto.pb $<
-	python ../generator/nanopb_generator.py fileproto.pb

example/client.c

@@ -1,116 +0,0 @@
-/* This is a simple TCP client that connects to port 1234 and prints a list
- * of files in a given directory.
- *
- * It directly deserializes and serializes messages from network, minimizing
- * memory use.
- * 
- * For flexibility, this example is implemented using posix api.
- * In a real embedded system you would typically use some other kind of
- * a communication and filesystem layer.
- */
-
-#include <sys/socket.h>
-#include <sys/types.h>
-#include <netinet/in.h>
-#include <unistd.h>
-#include <dirent.h>
-#include <stdio.h>
-#include <string.h>
-
-#include <pb_encode.h>
-#include <pb_decode.h>
-
-#include "fileproto.pb.h"
-#include "common.h"
-
-bool printfile_callback(pb_istream_t *stream, const pb_field_t *field, void **arg)
-{
-    FileInfo fileinfo;
-    
-    if (!pb_decode(stream, FileInfo_fields, &fileinfo))
-        return false;
-    
-    printf("%-10lld %s\n", (long long)fileinfo.inode, fileinfo.name);
-    
-    return true;
-}
-
-bool listdir(int fd, char *path)
-{
-    ListFilesRequest request;
-    ListFilesResponse response;
-    pb_istream_t input = pb_istream_from_socket(fd);
-    pb_ostream_t output = pb_ostream_from_socket(fd);
-    uint8_t zero = 0;
-    
-    if (path == NULL)
-    {
-        request.has_path = false;
-    }
-    else
-    {
-        request.has_path = true;
-        if (strlen(path) + 1 > sizeof(request.path))
-        {
-            fprintf(stderr, "Too long path.\n");
-            return false;
-        }
-        
-        strcpy(request.path, path);
-    }
-    
-    if (!pb_encode(&output, ListFilesRequest_fields, &request))
-    {
-        fprintf(stderr, "Encoding failed.\n");
-        return false;
-    }
-    
-    /* We signal the end of request with a 0 tag. */
-    pb_write(&output, &zero, 1);
-    
-    response.file.funcs.decode = &printfile_callback;
-    
-    if (!pb_decode(&input, ListFilesResponse_fields, &response))
-    {
-        fprintf(stderr, "Decode failed: %s\n", PB_GET_ERROR(&input));
-        return false;
-    }
-    
-    if (response.path_error)
-    {
-        fprintf(stderr, "Server reported error.\n");
-        return false;
-    }
-    
-    return true;
-}
-
-int main(int argc, char **argv)
-{
-    int sockfd;
-    struct sockaddr_in servaddr;
-    char *path = NULL;
-    
-    if (argc > 1)
-        path = argv[1];
-    
-    sockfd = socket(AF_INET, SOCK_STREAM, 0);
-    
-    memset(&servaddr, 0, sizeof(servaddr));
-    servaddr.sin_family = AF_INET;
-    servaddr.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
-    servaddr.sin_port = htons(1234);
-    
-    if (connect(sockfd, (struct sockaddr *)&servaddr, sizeof(servaddr)) != 0)
-    {
-        perror("connect");
-        return 1;
-    }
-    
-    if (!listdir(sockfd, path))
-        return 2;
-    
-    close(sockfd);
-    
-    return 0;
-}

example_avr_double/Makefile

@@ -1,22 +0,0 @@
-CFLAGS=-Wall -Werror -I .. -g -O0
-DEPS=double_conversion.c ../pb_decode.c ../pb_decode.h ../pb_encode.c ../pb_encode.h ../pb.h
-
-all: run_tests
-
-clean:
-	rm -f test_conversions encode_double decode_double doubleproto.pb.c doubleproto.pb.h
-
-test_conversions: test_conversions.c double_conversion.c
-	$(CC) $(CFLAGS) -o $@ $^
-
-%: %.c $(DEPS) doubleproto.pb.h doubleproto.pb.c
-	$(CC) $(CFLAGS) -o $@ $< double_conversion.c ../pb_decode.c ../pb_encode.c doubleproto.pb.c
-
-doubleproto.pb.c doubleproto.pb.h: doubleproto.proto ../generator/nanopb_generator.py
-	protoc -I. -I../generator -I/usr/include -odoubleproto.pb $<
-	python ../generator/nanopb_generator.py doubleproto.pb
-
-run_tests: test_conversions encode_double decode_double
-	./test_conversions
-	./encode_double | ./decode_double
-    

example_unions/Makefile

@@ -1,17 +0,0 @@
-CFLAGS=-ansi -Wall -Werror -I .. -g -O0
-DEPS=../pb_decode.c ../pb_decode.h ../pb_encode.c ../pb_encode.h ../pb.h
-
-all: encode decode
-	./encode 1 | ./decode
-	./encode 2 | ./decode
-	./encode 3 | ./decode
-
-clean:
-	rm -f encode unionproto.pb.h unionproto.pb.c
-
-%: %.c $(DEPS) unionproto.pb.h unionproto.pb.c
-	$(CC) $(CFLAGS) -o $@ $< ../pb_decode.c ../pb_encode.c unionproto.pb.c
-
-unionproto.pb.h unionproto.pb.c: unionproto.proto ../generator/nanopb_generator.py
-	protoc -I. -I../generator -I/usr/include -ounionproto.pb $<
-	python ../generator/nanopb_generator.py unionproto.pb

examples/network_server/Makefile

@@ -0,0 +1,17 @@
+# Include the nanopb provided Makefile rules
+include ../../extra/nanopb.mk
+
+# Compiler flags to enable all warnings & debug info
+CFLAGS = -ansi -Wall -Werror -g -O0
+CFLAGS += -I$(NANOPB_DIR)
+
+all: server client
+
+.SUFFIXES:
+
+clean:
+	rm -f server client fileproto.pb.c fileproto.pb.h
+
+%: %.c common.c fileproto.pb.c
+	$(CC) $(CFLAGS) -o $@ $^ $(NANOPB_CORE)
+

examples/network_server/README.txt

@@ -0,0 +1,60 @@
+Nanopb example "network_server"
+===============================
+
+This example demonstrates the use of nanopb to communicate over network
+connections. It consists of a server that sends file listings, and of
+a client that requests the file list from the server.
+
+Example usage
+-------------
+
+user@host:~/nanopb/examples/network_server$ make        # Build the example
+protoc -ofileproto.pb fileproto.proto
+python ../../generator/nanopb_generator.py fileproto.pb
+Writing to fileproto.pb.h and fileproto.pb.c
+cc -ansi -Wall -Werror -I .. -g -O0 -I../.. -o server server.c
+    ../../pb_decode.c ../../pb_encode.c fileproto.pb.c common.c
+cc -ansi -Wall -Werror -I .. -g -O0 -I../.. -o client client.c
+    ../../pb_decode.c ../../pb_encode.c fileproto.pb.c common.c
+
+user@host:~/nanopb/examples/network_server$ ./server &  # Start the server on background
+[1] 24462
+
+petteri@oddish:~/nanopb/examples/network_server$ ./client /bin  # Request the server to list /bin
+Got connection.
+Listing directory: /bin
+1327119    bzdiff
+1327126    bzless
+1327147    ps
+1327178    ntfsmove
+1327271    mv
+1327187    mount
+1327259    false
+1327266    tempfile
+1327285    zfgrep
+1327165    gzexe
+1327204    nc.openbsd
+1327260    uname
+
+
+Details of implementation
+-------------------------
+fileproto.proto contains the portable Google Protocol Buffers protocol definition.
+It could be used as-is to implement a server or a client in any other language, for
+example Python or Java.
+
+fileproto.options contains the nanopb-specific options for the protocol file. This
+sets the amount of space allocated for file names when decoding messages.
+
+common.c/h contains functions that allow nanopb to read and write directly from
+network socket. This way there is no need to allocate a separate buffer to store
+the message.
+
+server.c contains the code to open a listening socket, to respond to clients and
+to list directory contents.
+
+client.c contains the code to connect to a server, to send a request and to print
+the response message.
+
+The code is implemented using the POSIX socket api, but it should be easy enough
+to port into any other socket api, such as lwip.

examples/network_server/client.c

@@ -0,0 +1,142 @@
+/* This is a simple TCP client that connects to port 1234 and prints a list
+ * of files in a given directory.
+ *
+ * It directly deserializes and serializes messages from network, minimizing
+ * memory use.
+ * 
+ * For flexibility, this example is implemented using posix api.
+ * In a real embedded system you would typically use some other kind of
+ * a communication and filesystem layer.
+ */
+
+#include <sys/socket.h>
+#include <sys/types.h>
+#include <netinet/in.h>
+#include <unistd.h>
+#include <dirent.h>
+#include <stdio.h>
+#include <string.h>
+
+#include <pb_encode.h>
+#include <pb_decode.h>
+
+#include "fileproto.pb.h"
+#include "common.h"
+
+/* This callback function will be called once for each filename received
+ * from the server. The filenames will be printed out immediately, so that
+ * no memory has to be allocated for them.
+ */
+bool printfile_callback(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    FileInfo fileinfo = {};
+    
+    if (!pb_decode(stream, FileInfo_fields, &fileinfo))
+        return false;
+    
+    printf("%-10lld %s\n", (long long)fileinfo.inode, fileinfo.name);
+    
+    return true;
+}
+
+/* This function sends a request to socket 'fd' to list the files in
+ * directory given in 'path'. The results received from server will
+ * be printed to stdout.
+ */
+bool listdir(int fd, char *path)
+{
+    /* Construct and send the request to server */
+    {
+        ListFilesRequest request = {};
+        pb_ostream_t output = pb_ostream_from_socket(fd);
+        uint8_t zero = 0;
+        
+        /* In our protocol, path is optional. If it is not given,
+         * the server will list the root directory. */
+        if (path == NULL)
+        {
+            request.has_path = false;
+        }
+        else
+        {
+            request.has_path = true;
+            if (strlen(path) + 1 > sizeof(request.path))
+            {
+                fprintf(stderr, "Too long path.\n");
+                return false;
+            }
+            
+            strcpy(request.path, path);
+        }
+        
+        /* Encode the request. It is written to the socket immediately
+         * through our custom stream. */
+        if (!pb_encode(&output, ListFilesRequest_fields, &request))
+        {
+            fprintf(stderr, "Encoding failed: %s\n", PB_GET_ERROR(&output));
+            return false;
+        }
+        
+        /* We signal the end of request with a 0 tag. */
+        pb_write(&output, &zero, 1);
+    }
+    
+    /* Read back the response from server */
+    {
+        ListFilesResponse response = {};
+        pb_istream_t input = pb_istream_from_socket(fd);
+        
+        /* Give a pointer to our callback function, which will handle the
+         * filenames as they arrive. */
+        response.file.funcs.decode = &printfile_callback;
+        
+        if (!pb_decode(&input, ListFilesResponse_fields, &response))
+        {
+            fprintf(stderr, "Decode failed: %s\n", PB_GET_ERROR(&input));
+            return false;
+        }
+        
+        /* If the message from server decodes properly, but directory was
+         * not found on server side, we get path_error == true. */
+        if (response.path_error)
+        {
+            fprintf(stderr, "Server reported error.\n");
+            return false;
+        }
+    }
+    
+    return true;
+}
+
+int main(int argc, char **argv)
+{
+    int sockfd;
+    struct sockaddr_in servaddr;
+    char *path = NULL;
+    
+    if (argc > 1)
+        path = argv[1];
+    
+    sockfd = socket(AF_INET, SOCK_STREAM, 0);
+    
+    /* Connect to server running on localhost:1234 */
+    memset(&servaddr, 0, sizeof(servaddr));
+    servaddr.sin_family = AF_INET;
+    servaddr.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
+    servaddr.sin_port = htons(1234);
+    
+    if (connect(sockfd, (struct sockaddr *)&servaddr, sizeof(servaddr)) != 0)
+    {
+        perror("connect");
+        return 1;
+    }
+    
+    /* Send the directory listing request */
+    if (!listdir(sockfd, path))
+        return 2;
+    
+    /* Close connection */
+    close(sockfd);
+    
+    return 0;
+}

examples/network_server/server.c

@@ -23,11 +23,16 @@
 #include "fileproto.pb.h"
 #include "common.h"
 
+/* This callback function will be called once during the encoding.
+ * It will write out any number of FileInfo entries, without consuming unnecessary memory.
+ * This is accomplished by fetching the filenames one at a time and encoding them
+ * immediately.
+ */
 bool listdir_callback(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
 {
     DIR *dir = (DIR*) *arg;
     struct dirent *file;
-    FileInfo fileinfo;
+    FileInfo fileinfo = {};
     
     while ((file = readdir(dir)) != NULL)
     {
@@ -35,9 +40,12 @@ bool listdir_callback(pb_ostream_t *stream, const pb_field_t *field, void * cons
         strncpy(fileinfo.name, file->d_name, sizeof(fileinfo.name));
         fileinfo.name[sizeof(fileinfo.name) - 1] = '\0';
         
+        /* This encodes the header for the field, based on the constant info
+         * from pb_field_t. */
         if (!pb_encode_tag_for_field(stream, field))
             return false;
         
+        /* This encodes the data for the field, based on our FileInfo structure. */
         if (!pb_encode_submessage(stream, FileInfo_fields, &fileinfo))
             return false;
     }
@@ -45,43 +53,59 @@ bool listdir_callback(pb_ostream_t *stream, const pb_field_t *field, void * cons
     return true;
 }
 
+/* Handle one arriving client connection.
+ * Clients are expected to send a ListFilesRequest, terminated by a '0'.
+ * Server will respond with a ListFilesResponse message.
+ */
 void handle_connection(int connfd)
 {
-    ListFilesRequest request;
-    ListFilesResponse response;
-    pb_istream_t input = pb_istream_from_socket(connfd);
-    pb_ostream_t output = pb_ostream_from_socket(connfd);
-    DIR *directory;
+    DIR *directory = NULL;
     
-    if (!pb_decode(&input, ListFilesRequest_fields, &request))
+    /* Decode the message from the client and open the requested directory. */
     {
-        printf("Decode failed: %s\n", PB_GET_ERROR(&input));
-        return;
-    }
-    
-    directory = opendir(request.path);
-    
-    printf("Listing directory: %s\n", request.path);
-    
-    if (directory == NULL)
-    {
-        perror("opendir");
+        ListFilesRequest request = {};
+        pb_istream_t input = pb_istream_from_socket(connfd);
         
-        response.has_path_error = true;
-        response.path_error = true;
-        response.file.funcs.encode = NULL;
-    }
-    else
-    {
-        response.has_path_error = false;
-        response.file.funcs.encode = &listdir_callback;
-        response.file.arg = directory;
+        if (!pb_decode(&input, ListFilesRequest_fields, &request))
+        {
+            printf("Decode failed: %s\n", PB_GET_ERROR(&input));
+            return;
+        }
+        
+        directory = opendir(request.path);
+        printf("Listing directory: %s\n", request.path);
     }
     
-    if (!pb_encode(&output, ListFilesResponse_fields, &response))
+    /* List the files in the directory and transmit the response to client */
     {
-        printf("Encoding failed.\n");
+        ListFilesResponse response = {};
+        pb_ostream_t output = pb_ostream_from_socket(connfd);
+        
+        if (directory == NULL)
+        {
+            perror("opendir");
+            
+            /* Directory was not found, transmit error status */
+            response.has_path_error = true;
+            response.path_error = true;
+            response.file.funcs.encode = NULL;
+        }
+        else
+        {
+            /* Directory was found, transmit filenames */
+            response.has_path_error = false;
+            response.file.funcs.encode = &listdir_callback;
+            response.file.arg = directory;
+        }
+        
+        if (!pb_encode(&output, ListFilesResponse_fields, &response))
+        {
+            printf("Encoding failed: %s\n", PB_GET_ERROR(&output));
+        }
     }
+    
+    if (directory != NULL)
+        closedir(directory);
 }
 
 int main(int argc, char **argv)
@@ -90,8 +114,8 @@ int main(int argc, char **argv)
     struct sockaddr_in servaddr;
     int reuse = 1;
     
+    /* Listen on localhost:1234 for TCP connections */
     listenfd = socket(AF_INET, SOCK_STREAM, 0);
-    
     setsockopt(listenfd, SOL_SOCKET, SO_REUSEADDR, &reuse, sizeof(reuse));
     
     memset(&servaddr, 0, sizeof(servaddr));
@@ -112,6 +136,7 @@ int main(int argc, char **argv)
     
     for(;;)
     {
+        /* Wait for a client */
         connfd = accept(listenfd, NULL, NULL);
         
         if (connfd < 0)
@@ -128,4 +153,6 @@ int main(int argc, char **argv)
         
         close(connfd);
     }
+    
+    return 0;
 }

examples/simple/Makefile

@@ -0,0 +1,22 @@
+# Include the nanopb provided Makefile rules
+include ../../extra/nanopb.mk
+
+# Compiler flags to enable all warnings & debug info
+CFLAGS = -Wall -Werror -g -O0
+CFLAGS += -I$(NANOPB_DIR)
+
+# C source code files that are required
+CSRC  = simple.c                   # The main program
+CSRC += simple.pb.c                # The compiled protocol definition
+CSRC += $(NANOPB_DIR)/pb_encode.c  # The nanopb encoder
+CSRC += $(NANOPB_DIR)/pb_decode.c  # The nanopb decoder
+CSRC += $(NANOPB_DIR)/pb_common.c  # The nanopb common parts
+
+# Build rule for the main program
+simple: $(CSRC)
+	$(CC) $(CFLAGS) -osimple $(CSRC)
+
+# Build rule for the protocol
+simple.pb.c: simple.proto
+	$(PROTOC) $(PROTOC_OPTS) --nanopb_out=. simple.proto
+

examples/simple/README.txt

@@ -0,0 +1,29 @@
+Nanopb example "simple"
+=======================
+
+This example demonstrates the very basic use of nanopb. It encodes and
+decodes a simple message.
+
+The code uses four different API functions:
+
+  * pb_ostream_from_buffer() to declare the output buffer that is to be used
+  * pb_encode() to encode a message
+  * pb_istream_from_buffer() to declare the input buffer that is to be used
+  * pb_decode() to decode a message
+
+Example usage
+-------------
+
+On Linux, simply type "make" to build the example. After that, you can
+run it with the command: ./simple
+
+On other platforms, you first have to compile the protocol definition using
+the following command::
+
+  ../../generator-bin/protoc --nanopb_out=. simple.proto
+
+After that, add the following four files to your project and compile:
+
+  simple.c  simple.pb.c  pb_encode.c  pb_decode.c
+
+

examples/simple/simple.c

@@ -0,0 +1,68 @@
+#include <stdio.h>
+#include <pb_encode.h>
+#include <pb_decode.h>
+#include "simple.pb.h"
+
+int main()
+{
+    /* This is the buffer where we will store our message. */
+    uint8_t buffer[128];
+    size_t message_length;
+    bool status;
+    
+    /* Encode our message */
+    {
+        /* Allocate space on the stack to store the message data.
+         *
+         * Nanopb generates simple struct definitions for all the messages.
+         * - check out the contents of simple.pb.h! */
+        SimpleMessage message;
+        
+        /* Create a stream that will write to our buffer. */
+        pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
+        
+        /* Fill in the lucky number */
+        message.lucky_number = 13;
+        
+        /* Now we are ready to encode the message! */
+        status = pb_encode(&stream, SimpleMessage_fields, &message);
+        message_length = stream.bytes_written;
+        
+        /* Then just check for any errors.. */
+        if (!status)
+        {
+            printf("Encoding failed: %s\n", PB_GET_ERROR(&stream));
+            return 1;
+        }
+    }
+    
+    /* Now we could transmit the message over network, store it in a file or
+     * wrap it to a pigeon's leg.
+     */
+
+    /* But because we are lazy, we will just decode it immediately. */
+    
+    {
+        /* Allocate space for the decoded message. */
+        SimpleMessage message;
+        
+        /* Create a stream that reads from the buffer. */
+        pb_istream_t stream = pb_istream_from_buffer(buffer, message_length);
+        
+        /* Now we are ready to decode the message. */
+        status = pb_decode(&stream, SimpleMessage_fields, &message);
+        
+        /* Check for errors... */
+        if (!status)
+        {
+            printf("Decoding failed: %s\n", PB_GET_ERROR(&stream));
+            return 1;
+        }
+        
+        /* Print the data contained in the message. */
+        printf("Your lucky number was %d!\n", message.lucky_number);
+    }
+    
+    return 0;
+}
+

examples/simple/simple.proto

@@ -0,0 +1,7 @@
+// A very simple protocol definition, consisting of only
+// one message.
+
+message SimpleMessage {
+    required int32 lucky_number = 1;
+}
+

examples/using_double_on_avr/Makefile

@@ -0,0 +1,24 @@
+# Include the nanopb provided Makefile rules
+include ../../extra/nanopb.mk
+
+# Compiler flags to enable all warnings & debug info
+CFLAGS = -Wall -Werror -g -O0
+CFLAGS += -I$(NANOPB_DIR)
+
+all: run_tests
+
+.SUFFIXES:
+
+clean:
+	rm -f test_conversions encode_double decode_double doubleproto.pb.c doubleproto.pb.h
+
+test_conversions: test_conversions.c double_conversion.c
+	$(CC) $(CFLAGS) -o $@ $^
+
+%: %.c double_conversion.c doubleproto.pb.c
+	$(CC) $(CFLAGS) -o $@ $^ $(NANOPB_CORE)
+
+run_tests: test_conversions encode_double decode_double
+	./test_conversions
+	./encode_double | ./decode_double
+    

examples/using_double_on_avr/README.txt

@@ -1,3 +1,6 @@
+Nanopb example "using_double_on_avr"
+====================================
+
 Some processors/compilers, such as AVR-GCC, do not support the double
 datatype. Instead, they have sizeof(double) == 4. Because protocol
 binary format uses the double encoding directly, this causes trouble
@@ -9,7 +12,7 @@ platforms. The file double_conversion.c provides functions that
 convert these values to/from floats, without relying on compiler
 support.
 
-To use this method, you need to make two modifications to your code:
+To use this method, you need to make some modifications to your code:
 
 1) Change all 'double' fields into 'fixed64' in the .proto.
 
@@ -17,6 +20,6 @@ To use this method, you need to make two modifications to your code:
 
 3) Whenever reading a 'double' field, use double_to_float().
 
-The conversion routines should be as accurate as the float datatype can
+The conversion routines are as accurate as the float datatype can
 be. Furthermore, they should handle all special values (NaN, inf, denormalized
 numbers) correctly. There are testcases in test_conversions.c.

examples/using_union_messages/Makefile

@@ -0,0 +1,20 @@
+# Include the nanopb provided Makefile rules
+include ../../extra/nanopb.mk
+
+# Compiler flags to enable all warnings & debug info
+CFLAGS = -ansi -Wall -Werror -g -O0
+CFLAGS += -I$(NANOPB_DIR)
+
+all: encode decode
+	./encode 1 | ./decode
+	./encode 2 | ./decode
+	./encode 3 | ./decode
+
+.SUFFIXES:
+
+clean:
+	rm -f encode unionproto.pb.h unionproto.pb.c
+
+%: %.c unionproto.pb.c
+	$(CC) $(CFLAGS) -o $@ $^ $(NANOPB_CORE)
+

examples/using_union_messages/README.txt

@@ -0,0 +1,52 @@
+Nanopb example "using_union_messages"
+=====================================
+
+Union messages is a common technique in Google Protocol Buffers used to
+represent a group of messages, only one of which is passed at a time.
+It is described in Google's documentation:
+https://developers.google.com/protocol-buffers/docs/techniques#union
+
+This directory contains an example on how to encode and decode union messages
+with minimal memory usage. Usually, nanopb would allocate space to store
+all of the possible messages at the same time, even though at most one of
+them will be used at a time.
+
+By using some of the lower level nanopb APIs, we can manually generate the
+top level message, so that we only need to allocate the one submessage that
+we actually want. Similarly when decoding, we can manually read the tag of
+the top level message, and only then allocate the memory for the submessage
+after we already know its type.
+
+
+Example usage
+-------------
+
+Type `make` to run the example. It will build it and run commands like
+following:
+
+./encode 1 | ./decode
+Got MsgType1: 42
+./encode 2 | ./decode
+Got MsgType2: true
+./encode 3 | ./decode
+Got MsgType3: 3 1415
+
+This simply demonstrates that the "decode" program has correctly identified
+the type of the received message, and managed to decode it.
+
+
+Details of implementation
+-------------------------
+
+unionproto.proto contains the protocol used in the example. It consists of
+three messages: MsgType1, MsgType2 and MsgType3, which are collected together
+into UnionMessage.
+
+encode.c takes one command line argument, which should be a number 1-3. It
+then fills in and encodes the corresponding message, and writes it to stdout.
+
+decode.c reads a UnionMessage from stdin. Then it calls the function
+decode_unionmessage_type() to determine the type of the message. After that,
+the corresponding message is decoded and the contents of it printed to the
+screen.
+

extra/FindNanopb.cmake

@@ -1,4 +1,5 @@
-# Locate and configure the nanopb library.
+# This is an example script for use with CMake projects for locating and configuring
+# the nanopb library.
 #
 # The following varialbes have to be set:
 #
@@ -148,7 +149,7 @@ function(NANOPB_GENERATE_CPP SRCS HDRS)
              "${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.pb.h"
       COMMAND python
       ARGS ${NANOPB_GENERATOR_EXECUTABLE} ${FIL_WE}.pb
-      DEPENDS ${FIL_WE}.pb
+      DEPENDS "${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.pb"
       COMMENT "Running nanopb generator on ${FIL_WE}.pb"
       VERBATIM )
   endforeach()
@@ -181,8 +182,8 @@ mark_as_advanced(NANOPB_INCLUDE_DIRS)
 # Find nanopb source files
 set(NANOPB_SRCS)
 set(NANOPB_HDRS)
-list(APPEND _nanopb_srcs pb_decode.c pb_encode.c)
-list(APPEND _nanopb_hdrs pb_decode.h pb_encode.h pb.h)
+list(APPEND _nanopb_srcs pb_decode.c pb_encode.c pb_common.c)
+list(APPEND _nanopb_hdrs pb_decode.h pb_encode.h pb_common.h pb.h)
 
 foreach(FIL ${_nanopb_srcs})
   find_file(${FIL}__nano_pb_file NAMES ${FIL} PATHS ${NANOPB_SRC_ROOT_FOLDER} ${NANOPB_INCLUDE_DIRS})

extra/nanopb.mk

@@ -0,0 +1,37 @@
+# This is an include file for Makefiles. It provides rules for building
+# .pb.c and .pb.h files out of .proto, as well the path to nanopb core.
+
+# Path to the nanopb root directory
+NANOPB_DIR := $(abspath $(dir $(lastword $(MAKEFILE_LIST)))../)
+
+# Files for the nanopb core
+NANOPB_CORE = $(NANOPB_DIR)/pb_encode.c $(NANOPB_DIR)/pb_decode.c $(NANOPB_DIR)/pb_common.c
+
+# Check if we are running on Windows
+ifdef windir
+WINDOWS = 1
+endif
+ifdef WINDIR
+WINDOWS = 1
+endif
+
+# Check whether to use binary version of nanopb_generator or the
+# system-supplied python interpreter.
+ifneq "$(wildcard $(NANOPB_DIR)/generator-bin)" ""
+	# Binary package
+	PROTOC = $(NANOPB_DIR)/generator-bin/protoc
+	PROTOC_OPTS = 
+else
+	# Source only or git checkout
+	PROTOC = protoc
+	ifdef WINDOWS
+		PROTOC_OPTS = --plugin=protoc-gen-nanopb=$(NANOPB_DIR)/generator/protoc-gen-nanopb.bat
+	else
+		PROTOC_OPTS = --plugin=protoc-gen-nanopb=$(NANOPB_DIR)/generator/protoc-gen-nanopb
+	endif
+endif
+
+# Rule for building .pb.c and .pb.h
+%.pb.c %.pb.h: %.proto $(wildcard %.options)
+	$(PROTOC) $(PROTOC_OPTS) --nanopb_out=. $<
+

extra/pb_syshdr.h

@@ -0,0 +1,104 @@
+/* This is an example of a header file for platforms/compilers that do
+ * not come with stdint.h/stddef.h/stdbool.h/string.h. To use it, define
+ * PB_SYSTEM_HEADER as "pb_syshdr.h", including the quotes, and add the
+ * extra folder to your include path.
+ *
+ * It is very likely that you will need to customize this file to suit
+ * your platform. For any compiler that supports C99, this file should
+ * not be necessary.
+ */
+
+#ifndef _PB_SYSHDR_H_
+#define _PB_SYSHDR_H_
+
+/* stdint.h subset */
+#ifdef HAVE_STDINT_H
+#include <stdint.h>
+#else
+/* You will need to modify these to match the word size of your platform. */
+typedef signed char int8_t;
+typedef unsigned char uint8_t;
+typedef signed short int16_t;
+typedef unsigned short uint16_t;
+typedef signed int int32_t;
+typedef unsigned int uint32_t;
+typedef signed long long int64_t;
+typedef unsigned long long uint64_t;
+#endif
+
+/* stddef.h subset */
+#ifdef HAVE_STDDEF_H
+#include <stddef.h>
+#else
+
+typedef uint32_t size_t;
+#define offsetof(st, m) ((size_t)(&((st *)0)->m))
+
+#ifndef NULL
+#define NULL 0
+#endif
+
+#endif
+
+/* stdbool.h subset */
+#ifdef HAVE_STDBOOL_H
+#include <stdbool.h>
+#else
+
+#ifndef __cplusplus
+typedef int bool;
+#define false 0
+#define true 1
+#endif
+
+#endif
+
+/* stdlib.h subset */
+#ifdef PB_ENABLE_MALLOC
+#ifdef HAVE_STDLIB_H
+#include <stdlib.h>
+#else
+void *realloc(void *ptr, size_t size);
+void free(void *ptr);
+#endif
+#endif
+
+/* string.h subset */
+#ifdef HAVE_STRING_H
+#include <string.h>
+#else
+
+/* Implementations are from the Public Domain C Library (PDCLib). */
+static size_t strlen( const char * s )
+{
+    size_t rc = 0;
+    while ( s[rc] )
+    {
+        ++rc;
+    }
+    return rc;
+}
+
+static void * memcpy( void *s1, const void *s2, size_t n )
+{
+    char * dest = (char *) s1;
+    const char * src = (const char *) s2;
+    while ( n-- )
+    {
+        *dest++ = *src++;
+    }
+    return s1;
+}
+
+static void * memset( void * s, int c, size_t n )
+{
+    unsigned char * p = (unsigned char *) s;
+    while ( n-- )
+    {
+        *p++ = (unsigned char) c;
+    }
+    return s;
+}
+#endif
+
+#endif

generator/Makefile

@@ -1,5 +0,0 @@
-nanopb_pb2.py: nanopb.proto
-	protoc --python_out=. -I /usr/include -I . nanopb.proto
-
-plugin_pb2.py: plugin.proto
-	protoc --python_out=. -I /usr/include -I . plugin.proto

generator/nanopb_generator.py

@@ -1,36 +1,42 @@
 #!/usr/bin/python
 
 '''Generate header file for nanopb from a ProtoBuf FileDescriptorSet.'''
-nanopb_version = "nanopb-0.2.3-dev"
+nanopb_version = "nanopb-0.3.1"
+
+import sys
+
+try:
+    # Add some dummy imports to keep packaging tools happy.
+    import google, distutils.util # bbfreeze seems to need these
+    import pkg_resources # pyinstaller / protobuf 2.5 seem to need these
+except:
+    # Don't care, we will error out later if it is actually important.
+    pass
 
 try:
-    import google.protobuf.descriptor_pb2 as descriptor
     import google.protobuf.text_format as text_format
+    import google.protobuf.descriptor_pb2 as descriptor
 except:
-    print
-    print "*************************************************************"
-    print "*** Could not import the Google protobuf Python libraries ***"
-    print "*** Try installing package 'python-protobuf' or similar.  ***"
-    print "*************************************************************"
-    print
+    sys.stderr.write('''
+         *************************************************************
+         *** Could not import the Google protobuf Python libraries ***
+         *** Try installing package 'python-protobuf' or similar.  ***
+         *************************************************************
+    ''' + '\n')
     raise
 
 try:
-    import nanopb_pb2
+    import proto.nanopb_pb2 as nanopb_pb2
+    import proto.plugin_pb2 as plugin_pb2
 except:
-    print
-    print "***************************************************************"
-    print "*** Could not import the precompiled nanopb_pb2.py.         ***"
-    print "*** Run 'make' in the 'generator' folder to update the file.***"
-    print "***************************************************************"
-    print
+    sys.stderr.write('''
+         ********************************************************************
+         *** Failed to import the protocol definitions for generator.     ***
+         *** You have to run 'make' in the nanopb/generator/proto folder. ***
+         ********************************************************************
+    ''' + '\n')
     raise
 
-
-
-
-
-
 # ---------------------------------------------------------------------------
 #                     Generation of single fields
 # ---------------------------------------------------------------------------
@@ -38,22 +44,22 @@ except:
 import time
 import os.path
 
-# Values are tuple (c type, pb type)
+# Values are tuple (c type, pb type, encoded size)
 FieldD = descriptor.FieldDescriptorProto
 datatypes = {
-    FieldD.TYPE_BOOL:       ('bool',     'BOOL'),
-    FieldD.TYPE_DOUBLE:     ('double',   'DOUBLE'),
-    FieldD.TYPE_FIXED32:    ('uint32_t', 'FIXED32'),
-    FieldD.TYPE_FIXED64:    ('uint64_t', 'FIXED64'),
-    FieldD.TYPE_FLOAT:      ('float',    'FLOAT'),
-    FieldD.TYPE_INT32:      ('int32_t',  'INT32'),
-    FieldD.TYPE_INT64:      ('int64_t',  'INT64'),
-    FieldD.TYPE_SFIXED32:   ('int32_t',  'SFIXED32'),
-    FieldD.TYPE_SFIXED64:   ('int64_t',  'SFIXED64'),
-    FieldD.TYPE_SINT32:     ('int32_t',  'SINT32'),
-    FieldD.TYPE_SINT64:     ('int64_t',  'SINT64'),
-    FieldD.TYPE_UINT32:     ('uint32_t', 'UINT32'),
-    FieldD.TYPE_UINT64:     ('uint64_t', 'UINT64')
+    FieldD.TYPE_BOOL:       ('bool',     'BOOL',        1),
+    FieldD.TYPE_DOUBLE:     ('double',   'DOUBLE',      8),
+    FieldD.TYPE_FIXED32:    ('uint32_t', 'FIXED32',     4),
+    FieldD.TYPE_FIXED64:    ('uint64_t', 'FIXED64',     8),
+    FieldD.TYPE_FLOAT:      ('float',    'FLOAT',       4),
+    FieldD.TYPE_INT32:      ('int32_t',  'INT32',      10),
+    FieldD.TYPE_INT64:      ('int64_t',  'INT64',      10),
+    FieldD.TYPE_SFIXED32:   ('int32_t',  'SFIXED32',    4),
+    FieldD.TYPE_SFIXED64:   ('int64_t',  'SFIXED64',    8),
+    FieldD.TYPE_SINT32:     ('int32_t',  'SINT32',      5),
+    FieldD.TYPE_SINT64:     ('int64_t',  'SINT64',     10),
+    FieldD.TYPE_UINT32:     ('uint32_t', 'UINT32',      5),
+    FieldD.TYPE_UINT64:     ('uint64_t', 'UINT64',     10)
 }
 
 class Names:
@@ -83,6 +89,55 @@ def names_from_type_name(type_name):
         raise NotImplementedError("Lookup of non-absolute type names is not supported")
     return Names(type_name[1:].split('.'))
 
+def varint_max_size(max_value):
+    '''Returns the maximum number of bytes a varint can take when encoded.'''
+    for i in range(1, 11):
+        if (max_value >> (i * 7)) == 0:
+            return i
+    raise ValueError("Value too large for varint: " + str(max_value))
+
+assert varint_max_size(0) == 1
+assert varint_max_size(127) == 1
+assert varint_max_size(128) == 2
+
+class EncodedSize:
+    '''Class used to represent the encoded size of a field or a message.
+    Consists of a combination of symbolic sizes and integer sizes.'''
+    def __init__(self, value = 0, symbols = []):
+        if isinstance(value, (str, Names)):
+            symbols = [str(value)]
+            value = 0
+        self.value = value
+        self.symbols = symbols
+    
+    def __add__(self, other):
+        if isinstance(other, (int, long)):
+            return EncodedSize(self.value + other, self.symbols)
+        elif isinstance(other, (str, Names)):
+            return EncodedSize(self.value, self.symbols + [str(other)])
+        elif isinstance(other, EncodedSize):
+            return EncodedSize(self.value + other.value, self.symbols + other.symbols)
+        else:
+            raise ValueError("Cannot add size: " + repr(other))
+
+    def __mul__(self, other):
+        if isinstance(other, (int, long)):
+            return EncodedSize(self.value * other, [str(other) + '*' + s for s in self.symbols])
+        else:
+            raise ValueError("Cannot multiply size: " + repr(other))
+
+    def __str__(self):
+        if not self.symbols:
+            return str(self.value)
+        else:
+            return '(' + str(self.value) + ' + ' + ' + '.join(self.symbols) + ')'
+
+    def upperlimit(self):
+        if not self.symbols:
+            return self.value
+        else:
+            return 2**32 - 1
+
 class Enum:
     def __init__(self, names, desc, enum_options):
         '''desc is EnumDescriptorProto'''
@@ -113,6 +168,8 @@ class Field:
         self.max_size = None
         self.max_count = None
         self.array_decl = ""
+        self.enc_size = None
+        self.ctype = None
         
         # Parse field options
         if field_options.HasField("max_size"):
@@ -139,33 +196,15 @@ class Field:
         else:
             raise NotImplementedError(desc.label)
         
-        # Decide the C data type to use in the struct.
-        if datatypes.has_key(desc.type):
-            self.ctype, self.pbtype = datatypes[desc.type]
-        elif desc.type == FieldD.TYPE_ENUM:
-            self.pbtype = 'ENUM'
-            self.ctype = names_from_type_name(desc.type_name)
-            if self.default is not None:
-                self.default = self.ctype + self.default
-        elif desc.type == FieldD.TYPE_STRING:
-            self.pbtype = 'STRING'
-            if self.max_size is None:
-                can_be_static = False
-            else:
-                self.ctype = 'char'
-                self.array_decl += '[%d]' % self.max_size
-        elif desc.type == FieldD.TYPE_BYTES:
-            self.pbtype = 'BYTES'
-            if self.max_size is None:
-                can_be_static = False
-            else:
-                self.ctype = self.struct_name + self.name + 't'
-        elif desc.type == FieldD.TYPE_MESSAGE:
-            self.pbtype = 'MESSAGE'
-            self.ctype = self.submsgname = names_from_type_name(desc.type_name)
-        else:
-            raise NotImplementedError(desc.type)
+        # Check if the field can be implemented with static allocation
+        # i.e. whether the data size is known.
+        if desc.type == FieldD.TYPE_STRING and self.max_size is None:
+            can_be_static = False
         
+        if desc.type == FieldD.TYPE_BYTES and self.max_size is None:
+            can_be_static = False
+        
+        # Decide how the field data will be allocated
         if field_options.type == nanopb_pb2.FT_DEFAULT:
             if can_be_static:
                 field_options.type = nanopb_pb2.FT_STATIC
@@ -173,64 +212,140 @@ class Field:
                 field_options.type = nanopb_pb2.FT_CALLBACK
         
         if field_options.type == nanopb_pb2.FT_STATIC and not can_be_static:
-            raise Exception("Field %s is defined as static, but max_size or max_count is not given." % self.name)
+            raise Exception("Field %s is defined as static, but max_size or "
+                            "max_count is not given." % self.name)
         
         if field_options.type == nanopb_pb2.FT_STATIC:
             self.allocation = 'STATIC'
+        elif field_options.type == nanopb_pb2.FT_POINTER:
+            self.allocation = 'POINTER'
         elif field_options.type == nanopb_pb2.FT_CALLBACK:
             self.allocation = 'CALLBACK'
-            self.ctype = 'pb_callback_t'
-            self.array_decl = ''
         else:
             raise NotImplementedError(field_options.type)
-    
+        
+        # Decide the C data type to use in the struct.
+        if datatypes.has_key(desc.type):
+            self.ctype, self.pbtype, self.enc_size = datatypes[desc.type]
+        elif desc.type == FieldD.TYPE_ENUM:
+            self.pbtype = 'ENUM'
+            self.ctype = names_from_type_name(desc.type_name)
+            if self.default is not None:
+                self.default = self.ctype + self.default
+            self.enc_size = 5 # protoc rejects enum values > 32 bits
+        elif desc.type == FieldD.TYPE_STRING:
+            self.pbtype = 'STRING'
+            self.ctype = 'char'
+            if self.allocation == 'STATIC':
+                self.ctype = 'char'
+                self.array_decl += '[%d]' % self.max_size
+                self.enc_size = varint_max_size(self.max_size) + self.max_size
+        elif desc.type == FieldD.TYPE_BYTES:
+            self.pbtype = 'BYTES'
+            if self.allocation == 'STATIC':
+                self.ctype = self.struct_name + self.name + 't'
+                self.enc_size = varint_max_size(self.max_size) + self.max_size
+            elif self.allocation == 'POINTER':
+                self.ctype = 'pb_bytes_array_t'
+        elif desc.type == FieldD.TYPE_MESSAGE:
+            self.pbtype = 'MESSAGE'
+            self.ctype = self.submsgname = names_from_type_name(desc.type_name)
+            self.enc_size = None # Needs to be filled in after the message type is available
+        else:
+            raise NotImplementedError(desc.type)
+        
     def __cmp__(self, other):
         return cmp(self.tag, other.tag)
     
     def __str__(self):
-        if self.rules == 'OPTIONAL' and self.allocation == 'STATIC':
-            result = '    bool has_' + self.name + ';\n'
-        elif self.rules == 'REPEATED' and self.allocation == 'STATIC':
-            result = '    size_t ' + self.name + '_count;\n'
+        result = ''
+        if self.allocation == 'POINTER':
+            if self.rules == 'REPEATED':
+                result += '    pb_size_t ' + self.name + '_count;\n'
+            
+            if self.pbtype == 'MESSAGE':
+                # Use struct definition, so recursive submessages are possible
+                result += '    struct _%s *%s;' % (self.ctype, self.name)
+            elif self.rules == 'REPEATED' and self.pbtype in ['STRING', 'BYTES']:
+                # String/bytes arrays need to be defined as pointers to pointers
+                result += '    %s **%s;' % (self.ctype, self.name)
+            else:
+                result += '    %s *%s;' % (self.ctype, self.name)
+        elif self.allocation == 'CALLBACK':
+            result += '    pb_callback_t %s;' % self.name
         else:
-            result = ''
-        result += '    %s %s%s;' % (self.ctype, self.name, self.array_decl)
+            if self.rules == 'OPTIONAL' and self.allocation == 'STATIC':
+                result += '    bool has_' + self.name + ';\n'
+            elif self.rules == 'REPEATED' and self.allocation == 'STATIC':
+                result += '    pb_size_t ' + self.name + '_count;\n'
+            result += '    %s %s%s;' % (self.ctype, self.name, self.array_decl)
         return result
     
     def types(self):
         '''Return definitions for any special types this field might need.'''
         if self.pbtype == 'BYTES' and self.allocation == 'STATIC':
-            result = 'typedef struct {\n'
-            result += '    size_t size;\n'
-            result += '    uint8_t bytes[%d];\n' % self.max_size
-            result += '} %s;\n' % self.ctype
+            result = 'typedef PB_BYTES_ARRAY_T(%d) %s;\n' % (self.max_size, self.ctype)
         else:
             result = None
         return result
     
+    def get_initializer(self, null_init):
+        '''Return literal expression for this field's default value.'''
+        
+        if self.pbtype == 'MESSAGE':
+            if null_init:
+                return '%s_init_zero' % self.ctype
+            else:
+                return '%s_init_default' % self.ctype
+        
+        if self.default is None or null_init:
+            if self.pbtype == 'STRING':
+                return '""'
+            elif self.pbtype == 'BYTES':
+                return '{0, {0}}'
+            elif self.pbtype == 'ENUM':
+                return '(%s)0' % self.ctype
+            else:
+                return '0'
+        
+        default = str(self.default)
+        
+        if self.pbtype == 'STRING':
+            default = default.encode('utf-8').encode('string_escape')
+            default = default.replace('"', '\\"')
+            default = '"' + default + '"'
+        elif self.pbtype == 'BYTES':
+            data = default.decode('string_escape')
+            data = ['0x%02x' % ord(c) for c in data]
+            if len(data) == 0:
+                default = '{0, {0}}'
+            else:
+                default = '{%d, {%s}}' % (len(data), ','.join(data))
+        elif self.pbtype in ['FIXED32', 'UINT32']:
+            default += 'u'
+        elif self.pbtype in ['FIXED64', 'UINT64']:
+            default += 'ull'
+        elif self.pbtype in ['SFIXED64', 'INT64']:
+            default += 'll'
+        
+        return default
+    
     def default_decl(self, declaration_only = False):
         '''Return definition for this field's default value.'''
         if self.default is None:
             return None
 
-        ctype, default = self.ctype, self.default
+        ctype = self.ctype
+        default = self.get_initializer(False)
         array_decl = ''
         
         if self.pbtype == 'STRING':
             if self.allocation != 'STATIC':
                 return None # Not implemented
-        
             array_decl = '[%d]' % self.max_size
-            default = str(self.default).encode('string_escape')
-            default = default.replace('"', '\\"')
-            default = '"' + default + '"'
         elif self.pbtype == 'BYTES':
             if self.allocation != 'STATIC':
                 return None # Not implemented
-
-            data = self.default.decode('string_escape')
-            data = ['0x%02x' % ord(c) for c in data]
-            default = '{%d, {%s}}' % (len(data), ','.join(data))
         
         if declaration_only:
             return 'extern const %s %s_default%s;' % (ctype, self.struct_name + self.name, array_decl)
@@ -246,14 +361,14 @@ class Field:
         '''Return the pb_field_t initializer to use in the constant array.
         prev_field_name is the name of the previous field or None.
         '''
-        result  = '    PB_FIELD2(%3d, ' % self.tag
+        result  = '    PB_FIELD(%3d, ' % self.tag
         result += '%-8s, ' % self.pbtype
         result += '%s, ' % self.rules
-        result += '%s, ' % self.allocation
+        result += '%-8s, ' % self.allocation
+        result += '%s, ' % ("FIRST" if not prev_field_name else "OTHER")
         result += '%s, ' % self.struct_name
         result += '%s, ' % self.name
         result += '%s, ' % (prev_field_name or self.name)
-        result += '%s, ' % ("first" if not prev_field_name else "other")
         
         if self.pbtype == 'MESSAGE':
             result += '&%s_fields)' % self.submsgname
@@ -261,6 +376,8 @@ class Field:
             result += '0)'
         elif self.pbtype in ['BYTES', 'STRING'] and self.allocation != 'STATIC':
             result += '0)' # Arbitrary size default values not implemented
+        elif self.rules == 'OPTEXT':
+            result += '0)' # Default value for extensions is not implemented
         else:
             result += '&%s_default)' % (self.struct_name + self.name)
         
@@ -277,6 +394,51 @@ class Field:
 
         return max(self.tag, self.max_size, self.max_count)        
 
+    def encoded_size(self, allmsgs):
+        '''Return the maximum size that this field can take when encoded,
+        including the field tag. If the size cannot be determined, returns
+        None.'''
+        
+        if self.allocation != 'STATIC':
+            return None
+        
+        if self.pbtype == 'MESSAGE':
+            for msg in allmsgs:
+                if msg.name == self.submsgname:
+                    encsize = msg.encoded_size(allmsgs)
+                    if encsize is None:
+                        return None # Submessage size is indeterminate
+                        
+                    # Include submessage length prefix
+                    encsize += varint_max_size(encsize.upperlimit())
+                    break
+            else:
+                # Submessage cannot be found, this currently occurs when
+                # the submessage type is defined in a different file.
+                # Instead of direct numeric value, reference the size that
+                # has been #defined in the other file.
+                encsize = EncodedSize(self.submsgname + 'size')
+
+                # We will have to make a conservative assumption on the length
+                # prefix size, though.
+                encsize += 5
+
+        elif self.enc_size is None:
+            raise RuntimeError("Could not determine encoded size for %s.%s"
+                               % (self.struct_name, self.name))
+        else:
+            encsize = EncodedSize(self.enc_size)
+        
+        encsize += varint_max_size(self.tag << 3) # Tag + wire type
+
+        if self.rules == 'REPEATED':
+            # Decoders must be always able to handle unpacked arrays.
+            # Therefore we have to reserve space for it, even though
+            # we emit packed arrays ourselves.
+            encsize *= self.max_count
+        
+        return encsize
+
 
 class ExtensionRange(Field):
     def __init__(self, struct_name, range_start, field_options):
@@ -305,6 +467,12 @@ class ExtensionRange(Field):
     
     def tags(self):
         return ''
+    
+    def encoded_size(self, allmsgs):
+        # We exclude extensions from the count, because they cannot be known
+        # until runtime. Other option would be to return None here, but this
+        # way the value remains useful if extensions are not used.
+        return EncodedSize(0)
 
 class ExtensionField(Field):
     def __init__(self, struct_name, desc, field_options):
@@ -318,6 +486,11 @@ class ExtensionField(Field):
             self.skip = False
             self.rules = 'OPTEXT'
 
+    def tags(self):
+        '''Return the #define for the tag number of this field.'''
+        identifier = '%s_tag' % self.fullname
+        return '#define %-40s %d\n' % (identifier, self.tag)
+
     def extension_decl(self):
         '''Declaration of the extension type in the .pb.h file'''
         if self.skip:
@@ -373,7 +546,7 @@ class Message:
 
     def get_dependencies(self):
         '''Get list of type names that this structure refers to.'''
-        return [str(field.ctype) for field in self.fields]
+        return [str(field.ctype) for field in self.fields if field.allocation == 'STATIC']
     
     def __str__(self):
         result = 'typedef struct _%s {\n' % self.name
@@ -405,6 +578,34 @@ class Message:
                 result += types + '\n'
         return result
     
+    def get_initializer(self, null_init):
+        if not self.ordered_fields:
+            return '{0}'
+    
+        parts = []
+        for field in self.ordered_fields:
+            if field.allocation == 'STATIC':
+                if field.rules == 'REPEATED':
+                    parts.append('0')
+                    parts.append('{'
+                                 + ', '.join([field.get_initializer(null_init)] * field.max_count)
+                                 + '}')
+                elif field.rules == 'OPTIONAL':
+                    parts.append('false')
+                    parts.append(field.get_initializer(null_init))
+                else:
+                    parts.append(field.get_initializer(null_init))
+            elif field.allocation == 'POINTER':
+                if field.rules == 'REPEATED':
+                    parts.append('0')
+                parts.append('NULL')
+            elif field.allocation == 'CALLBACK':
+                if field.pbtype == 'EXTENSION':
+                    parts.append('NULL')
+                else:
+                    parts.append('{{NULL}, NULL}')
+        return '{' + ', '.join(parts) + '}'
+    
     def default_decl(self, declaration_only = False):
         result = ""
         for field in self.fields:
@@ -429,6 +630,18 @@ class Message:
         result += '    PB_LAST_FIELD\n};'
         return result
 
+    def encoded_size(self, allmsgs):
+        '''Return the maximum size that this message can take when encoded.
+        If the size cannot be determined, returns None.
+        '''
+        size = EncodedSize(0)
+        for field in self.fields:
+            fsize = field.encoded_size(allmsgs)
+            if fsize is None:
+                return None
+            size += fsize
+        
+        return size
 
 
 # ---------------------------------------------------------------------------
@@ -479,13 +692,17 @@ def parse_file(fdesc, file_options):
     
     for names, message in iterate_messages(fdesc, base_name):
         message_options = get_nanopb_suboptions(message, file_options, names)
+        
+        if message_options.skip_message:
+            continue
+        
         messages.append(Message(names, message, message_options))
         for enum in message.enum_type:
             enum_options = get_nanopb_suboptions(enum, message_options, names + enum.name)
             enums.append(Enum(names, enum, enum_options))
     
     for names, extension in iterate_extensions(fdesc, base_name):
-        field_options = get_nanopb_suboptions(extension, file_options, names)
+        field_options = get_nanopb_suboptions(extension, file_options, names + extension.name)
         if field_options.type != nanopb_pb2.FT_IGNORE:
             extensions.append(ExtensionField(names, extension, field_options))
     
@@ -547,11 +764,14 @@ def generate_header(dependencies, headername, enums, messages, extensions, optio
     '''
     
     yield '/* Automatically generated nanopb header */\n'
-    yield '/* Generated by %s at %s. */\n\n' % (nanopb_version, time.asctime())
+    if options.notimestamp:
+        yield '/* Generated by %s */\n\n' % (nanopb_version)
+    else:
+        yield '/* Generated by %s at %s. */\n\n' % (nanopb_version, time.asctime())
     
     symbol = make_identifier(headername)
-    yield '#ifndef _PB_%s_\n' % symbol
-    yield '#define _PB_%s_\n' % symbol
+    yield '#ifndef PB_%s_INCLUDED\n' % symbol
+    yield '#define PB_%s_INCLUDED\n' % symbol
     try:
         yield options.libformat % ('pb.h')
     except TypeError:
@@ -561,9 +781,14 @@ def generate_header(dependencies, headername, enums, messages, extensions, optio
     
     for dependency in dependencies:
         noext = os.path.splitext(dependency)[0]
-        yield options.genformat % (noext + '.' + options.extension + '.h')
+        yield options.genformat % (noext + options.extension + '.h')
         yield '\n'
 
+    yield '#if PB_PROTO_HEADER_VERSION != 30\n'
+    yield '#error Regenerate this file with the current version of nanopb generator.\n'
+    yield '#endif\n'
+    yield '\n'
+
     yield '#ifdef __cplusplus\n'
     yield 'extern "C" {\n'
     yield '#endif\n\n'
@@ -588,17 +813,37 @@ def generate_header(dependencies, headername, enums, messages, extensions, optio
         yield msg.default_decl(True)
     yield '\n'
     
+    yield '/* Initializer values for message structs */\n'
+    for msg in messages:
+        identifier = '%s_init_default' % msg.name
+        yield '#define %-40s %s\n' % (identifier, msg.get_initializer(False))
+    for msg in messages:
+        identifier = '%s_init_zero' % msg.name
+        yield '#define %-40s %s\n' % (identifier, msg.get_initializer(True))
+    yield '\n'
+    
     yield '/* Field tags (for use in manual encoding/decoding) */\n'
     for msg in sort_dependencies(messages):
         for field in msg.fields:
             yield field.tags()
+    for extension in extensions:
+        yield extension.tags()
     yield '\n'
     
     yield '/* Struct field encoding specification for nanopb */\n'
     for msg in messages:
         yield msg.fields_declaration() + '\n'
+    yield '\n'
     
-    yield '\n#ifdef __cplusplus\n'
+    yield '/* Maximum encoded size of messages (where known) */\n'
+    for msg in messages:
+        msize = msg.encoded_size(messages)
+        if msize is not None:
+            identifier = '%s_size' % msg.name
+            yield '#define %-40s %s\n' % (identifier, msize)
+    yield '\n'
+    
+    yield '#ifdef __cplusplus\n'
     yield '} /* extern "C" */\n'
     yield '#endif\n'
     
@@ -609,10 +854,18 @@ def generate_source(headername, enums, messages, extensions, options):
     '''Generate content for a source file.'''
     
     yield '/* Automatically generated nanopb constant definitions */\n'
-    yield '/* Generated by %s at %s. */\n\n' % (nanopb_version, time.asctime())
+    if options.notimestamp:
+        yield '/* Generated by %s */\n\n' % (nanopb_version)
+    else:
+        yield '/* Generated by %s at %s. */\n\n' % (nanopb_version, time.asctime())
     yield options.genformat % (headername)
     yield '\n'
     
+    yield '#if PB_PROTO_HEADER_VERSION != 30\n'
+    yield '#error Regenerate this file with the current version of nanopb generator.\n'
+    yield '#endif\n'
+    yield '\n'
+    
     for msg in messages:
         yield msg.default_decl(False)
     
@@ -653,16 +906,6 @@ def generate_source(headername, enums, messages, extensions, options):
     if worst > 255 or checks:
         yield '\n/* Check that field information fits in pb_field_t */\n'
         
-        if worst < 65536:
-            yield '#if !defined(PB_FIELD_16BIT) && !defined(PB_FIELD_32BIT)\n'
-            if worst > 255:
-                yield '#error Field descriptor for %s is too large. Define PB_FIELD_16BIT to fix this.\n' % worst_field
-            else:
-                assertion = ' && '.join(str(c) + ' < 256' for c in checks)
-                msgs = '_'.join(str(n) for n in checks_msgnames)
-                yield 'STATIC_ASSERT((%s), YOU_MUST_DEFINE_PB_FIELD_16BIT_FOR_MESSAGES_%s)\n'%(assertion,msgs)
-            yield '#endif\n\n'
-        
         if worst > 65535 or checks:
             yield '#if !defined(PB_FIELD_32BIT)\n'
             if worst > 65535:
@@ -670,8 +913,32 @@ def generate_source(headername, enums, messages, extensions, options):
             else:
                 assertion = ' && '.join(str(c) + ' < 65536' for c in checks)
                 msgs = '_'.join(str(n) for n in checks_msgnames)
-                yield 'STATIC_ASSERT((%s), YOU_MUST_DEFINE_PB_FIELD_32BIT_FOR_MESSAGES_%s)\n'%(assertion,msgs)
-            yield '#endif\n'
+                yield '/* If you get an error here, it means that you need to define PB_FIELD_32BIT\n'
+                yield ' * compile-time option. You can do that in pb.h or on compiler command line.\n'
+                yield ' * \n'
+                yield ' * The reason you need to do this is that some of your messages contain tag\n'
+                yield ' * numbers or field sizes that are larger than what can fit in 8 or 16 bit\n'
+                yield ' * field descriptors.\n'
+                yield ' */\n'
+                yield 'PB_STATIC_ASSERT((%s), YOU_MUST_DEFINE_PB_FIELD_32BIT_FOR_MESSAGES_%s)\n'%(assertion,msgs)
+            yield '#endif\n\n'
+        
+        if worst < 65536:
+            yield '#if !defined(PB_FIELD_16BIT) && !defined(PB_FIELD_32BIT)\n'
+            if worst > 255:
+                yield '#error Field descriptor for %s is too large. Define PB_FIELD_16BIT to fix this.\n' % worst_field
+            else:
+                assertion = ' && '.join(str(c) + ' < 256' for c in checks)
+                msgs = '_'.join(str(n) for n in checks_msgnames)
+                yield '/* If you get an error here, it means that you need to define PB_FIELD_16BIT\n'
+                yield ' * compile-time option. You can do that in pb.h or on compiler command line.\n'
+                yield ' * \n'
+                yield ' * The reason you need to do this is that some of your messages contain tag\n'
+                yield ' * numbers or field sizes that are larger than what can fit in the default\n'
+                yield ' * 8 bit descriptors.\n'
+                yield ' */\n'
+                yield 'PB_STATIC_ASSERT((%s), YOU_MUST_DEFINE_PB_FIELD_16BIT_FOR_MESSAGES_%s)\n'%(assertion,msgs)
+            yield '#endif\n\n'
     
     # Add check for sizeof(double)
     has_double = False
@@ -686,7 +953,7 @@ def generate_source(headername, enums, messages, extensions, options):
         yield ' * These are not directly supported by nanopb, but see example_avr_double.\n'
         yield ' * To get rid of this error, remove any double fields from your .proto.\n'
         yield ' */\n'
-        yield 'STATIC_ASSERT(sizeof(double) == 8, DOUBLE_MUST_BE_8_BYTES)\n'
+        yield 'PB_STATIC_ASSERT(sizeof(double) == 8, DOUBLE_MUST_BE_8_BYTES)\n'
     
     yield '\n'
 
@@ -717,6 +984,7 @@ class Globals:
     '''Ugly global variables, should find a good way to pass these.'''
     verbose_options = False
     separate_options = []
+    matched_namemasks = set()
 
 def get_nanopb_suboptions(subdesc, options, name):
     '''Get copy of options, and merge information from subdesc.'''
@@ -727,6 +995,7 @@ def get_nanopb_suboptions(subdesc, options, name):
     dotname = '.'.join(name.parts)
     for namemask, options in Globals.separate_options:
         if fnmatch(dotname, namemask):
+            Globals.matched_namemasks.add(namemask)
             new_options.MergeFrom(options)
     
     # Handle options defined in .proto
@@ -746,8 +1015,8 @@ def get_nanopb_suboptions(subdesc, options, name):
         new_options.MergeFrom(ext)
     
     if Globals.verbose_options:
-        print "Options for " + dotname + ":"
-        print text_format.MessageToString(new_options)
+        sys.stderr.write("Options for " + dotname + ": ")
+        sys.stderr.write(text_format.MessageToString(new_options) + "\n")
     
     return new_options
 
@@ -766,8 +1035,8 @@ optparser = OptionParser(
              "Output will be written to file.pb.h and file.pb.c.")
 optparser.add_option("-x", dest="exclude", metavar="FILE", action="append", default=[],
     help="Exclude file from generated #include list.")
-optparser.add_option("-e", "--extension", dest="extension", metavar="EXTENSION", default="pb",
-    help="Set extension to use instead of 'pb' for generated files. [default: %default]")
+optparser.add_option("-e", "--extension", dest="extension", metavar="EXTENSION", default=".pb",
+    help="Set extension to use instead of '.pb' for generated files. [default: %default]")
 optparser.add_option("-f", "--options-file", dest="options_file", metavar="FILE", default="%s.options",
     help="Set name of a separate generator options file.")
 optparser.add_option("-Q", "--generated-include-format", dest="genformat",
@@ -776,6 +1045,8 @@ optparser.add_option("-Q", "--generated-include-format", dest="genformat",
 optparser.add_option("-L", "--library-include-format", dest="libformat",
     metavar="FORMAT", default='#include <%s>\n',
     help="Set format string to use for including the nanopb pb.h header. [default: %default]")
+optparser.add_option("-T", "--no-timestamp", dest="notimestamp", action="store_true", default=False,
+    help="Don't add timestamp to .pb.h and .pb.c preambles")
 optparser.add_option("-q", "--quiet", dest="quiet", action="store_true", default=False,
     help="Don't print anything except errors.")
 optparser.add_option("-v", "--verbose", dest="verbose", action="store_true", default=False,
@@ -805,28 +1076,38 @@ def process_file(filename, fdesc, options):
         fdesc = descriptor.FileDescriptorSet.FromString(data).file[0]
     
     # Check if there is a separate .options file
+    had_abspath = False
     try:
         optfilename = options.options_file % os.path.splitext(filename)[0]
     except TypeError:
         # No %s specified, use the filename as-is
         optfilename = options.options_file
-    
-    if options.verbose:
-        print 'Reading options from ' + optfilename
-    
+        had_abspath = True
+
     if os.path.isfile(optfilename):
+        if options.verbose:
+            sys.stderr.write('Reading options from ' + optfilename + '\n')
+
         Globals.separate_options = read_options_file(open(optfilename, "rU"))
     else:
+        # If we are given a full filename and it does not exist, give an error.
+        # However, don't give error when we automatically look for .options file
+        # with the same name as .proto.
+        if options.verbose or had_abspath:
+            sys.stderr.write('Options file not found: ' + optfilename)
+
         Globals.separate_options = []
+
+    Globals.matched_namemasks = set()
     
     # Parse the file
     file_options = get_nanopb_suboptions(fdesc, toplevel_options, Names([filename]))
     enums, messages, extensions = parse_file(fdesc, file_options)
-    
+
     # Decide the file names
     noext = os.path.splitext(filename)[0]
-    headername = noext + '.' + options.extension + '.h'
-    sourcename = noext + '.' + options.extension + '.c'
+    headername = noext + options.extension + '.h'
+    sourcename = noext + options.extension + '.c'
     headerbasename = os.path.basename(headername)
     
     # List of .proto files that should not be included in the C header file
@@ -840,6 +1121,14 @@ def process_file(filename, fdesc, options):
     sourcedata = ''.join(generate_source(headerbasename, enums,
                                          messages, extensions, options))
 
+    # Check if there were any lines in .options that did not match a member
+    unmatched = [n for n,o in Globals.separate_options if n not in Globals.matched_namemasks]
+    if unmatched and not options.quiet:
+        sys.stderr.write("Following patterns in " + optfilename + " did not match any fields: "
+                         + ', '.join(unmatched) + "\n")
+        if not Globals.verbose_options:
+            sys.stderr.write("Use  protoc --nanopb-out=-v:.   to see a list of the field names.\n")
+
     return {'headername': headername, 'headerdata': headerdata,
             'sourcename': sourcename, 'sourcedata': sourcedata}
     
@@ -861,7 +1150,8 @@ def main_cli():
         results = process_file(filename, None, options)
         
         if not options.quiet:
-            print "Writing to " + results['headername'] + " and " + results['sourcename']
+            sys.stderr.write("Writing to " + results['headername'] + " and "
+                             + results['sourcename'] + "\n")
     
         open(results['headername'], 'w').write(results['headerdata'])
         open(results['sourcename'], 'w').write(results['sourcedata'])        
@@ -869,7 +1159,13 @@ def main_cli():
 def main_plugin():
     '''Main function when invoked as a protoc plugin.'''
 
-    import plugin_pb2
+    import sys
+    if sys.platform == "win32":
+        import os, msvcrt
+        # Set stdin and stdout to binary mode
+        msvcrt.setmode(sys.stdin.fileno(), os.O_BINARY)
+        msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)
+    
     data = sys.stdin.read()
     request = plugin_pb2.CodeGeneratorRequest.FromString(data)
     
@@ -877,10 +1173,7 @@ def main_plugin():
     args = shlex.split(request.parameter)
     options, dummy = optparser.parse_args(args)
     
-    # We can't go printing stuff to stdout
-    Globals.verbose_options = False
-    options.verbose = False
-    options.quiet = True
+    Globals.verbose_options = options.verbose
     
     response = plugin_pb2.CodeGeneratorResponse()
     
@@ -901,7 +1194,7 @@ def main_plugin():
 
 if __name__ == '__main__':
     # Check if we are running as a plugin under protoc
-    if 'protoc-gen-' in sys.argv[0]:
+    if 'protoc-gen-' in sys.argv[0] or '--protoc-plugin' in sys.argv:
         main_plugin()
     else:
         main_cli()

generator/nanopb_pb2.py

@@ -1,158 +0,0 @@
-# Generated by the protocol buffer compiler.  DO NOT EDIT!
-
-from google.protobuf import descriptor
-from google.protobuf import message
-from google.protobuf import reflection
-from google.protobuf import descriptor_pb2
-# @@protoc_insertion_point(imports)
-
-
-import google.protobuf.descriptor_pb2
-
-DESCRIPTOR = descriptor.FileDescriptor(
-  name='nanopb.proto',
-  package='',
-  serialized_pb='\n\x0cnanopb.proto\x1a google/protobuf/descriptor.proto\"\x92\x01\n\rNanoPBOptions\x12\x10\n\x08max_size\x18\x01 \x01(\x05\x12\x11\n\tmax_count\x18\x02 \x01(\x05\x12$\n\x04type\x18\x03 \x01(\x0e\x32\n.FieldType:\nFT_DEFAULT\x12\x18\n\nlong_names\x18\x04 \x01(\x08:\x04true\x12\x1c\n\rpacked_struct\x18\x05 \x01(\x08:\x05\x66\x61lse*J\n\tFieldType\x12\x0e\n\nFT_DEFAULT\x10\x00\x12\x0f\n\x0b\x46T_CALLBACK\x10\x01\x12\r\n\tFT_STATIC\x10\x02\x12\r\n\tFT_IGNORE\x10\x03:E\n\x0enanopb_fileopt\x12\x1c.google.protobuf.FileOptions\x18\xf2\x07 \x01(\x0b\x32\x0e.NanoPBOptions:G\n\rnanopb_msgopt\x12\x1f.google.protobuf.MessageOptions\x18\xf2\x07 \x01(\x0b\x32\x0e.NanoPBOptions:E\n\x0enanopb_enumopt\x12\x1c.google.protobuf.EnumOptions\x18\xf2\x07 \x01(\x0b\x32\x0e.NanoPBOptions:>\n\x06nanopb\x12\x1d.google.protobuf.FieldOptions\x18\xf2\x07 \x01(\x0b\x32\x0e.NanoPBOptions')
-
-_FIELDTYPE = descriptor.EnumDescriptor(
-  name='FieldType',
-  full_name='FieldType',
-  filename=None,
-  file=DESCRIPTOR,
-  values=[
-    descriptor.EnumValueDescriptor(
-      name='FT_DEFAULT', index=0, number=0,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='FT_CALLBACK', index=1, number=1,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='FT_STATIC', index=2, number=2,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='FT_IGNORE', index=3, number=3,
-      options=None,
-      type=None),
-  ],
-  containing_type=None,
-  options=None,
-  serialized_start=199,
-  serialized_end=273,
-)
-
-
-FT_DEFAULT = 0
-FT_CALLBACK = 1
-FT_STATIC = 2
-FT_IGNORE = 3
-
-NANOPB_FILEOPT_FIELD_NUMBER = 1010
-nanopb_fileopt = descriptor.FieldDescriptor(
-  name='nanopb_fileopt', full_name='nanopb_fileopt', index=0,
-  number=1010, type=11, cpp_type=10, label=1,
-  has_default_value=False, default_value=None,
-  message_type=None, enum_type=None, containing_type=None,
-  is_extension=True, extension_scope=None,
-  options=None)
-NANOPB_MSGOPT_FIELD_NUMBER = 1010
-nanopb_msgopt = descriptor.FieldDescriptor(
-  name='nanopb_msgopt', full_name='nanopb_msgopt', index=1,
-  number=1010, type=11, cpp_type=10, label=1,
-  has_default_value=False, default_value=None,
-  message_type=None, enum_type=None, containing_type=None,
-  is_extension=True, extension_scope=None,
-  options=None)
-NANOPB_ENUMOPT_FIELD_NUMBER = 1010
-nanopb_enumopt = descriptor.FieldDescriptor(
-  name='nanopb_enumopt', full_name='nanopb_enumopt', index=2,
-  number=1010, type=11, cpp_type=10, label=1,
-  has_default_value=False, default_value=None,
-  message_type=None, enum_type=None, containing_type=None,
-  is_extension=True, extension_scope=None,
-  options=None)
-NANOPB_FIELD_NUMBER = 1010
-nanopb = descriptor.FieldDescriptor(
-  name='nanopb', full_name='nanopb', index=3,
-  number=1010, type=11, cpp_type=10, label=1,
-  has_default_value=False, default_value=None,
-  message_type=None, enum_type=None, containing_type=None,
-  is_extension=True, extension_scope=None,
-  options=None)
-
-
-_NANOPBOPTIONS = descriptor.Descriptor(
-  name='NanoPBOptions',
-  full_name='NanoPBOptions',
-  filename=None,
-  file=DESCRIPTOR,
-  containing_type=None,
-  fields=[
-    descriptor.FieldDescriptor(
-      name='max_size', full_name='NanoPBOptions.max_size', index=0,
-      number=1, type=5, cpp_type=1, label=1,
-      has_default_value=False, default_value=0,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='max_count', full_name='NanoPBOptions.max_count', index=1,
-      number=2, type=5, cpp_type=1, label=1,
-      has_default_value=False, default_value=0,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='type', full_name='NanoPBOptions.type', index=2,
-      number=3, type=14, cpp_type=8, label=1,
-      has_default_value=True, default_value=0,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='long_names', full_name='NanoPBOptions.long_names', index=3,
-      number=4, type=8, cpp_type=7, label=1,
-      has_default_value=True, default_value=True,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='packed_struct', full_name='NanoPBOptions.packed_struct', index=4,
-      number=5, type=8, cpp_type=7, label=1,
-      has_default_value=True, default_value=False,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-  ],
-  extensions=[
-  ],
-  nested_types=[],
-  enum_types=[
-  ],
-  options=None,
-  is_extendable=False,
-  extension_ranges=[],
-  serialized_start=51,
-  serialized_end=197,
-)
-
-_NANOPBOPTIONS.fields_by_name['type'].enum_type = _FIELDTYPE
-DESCRIPTOR.message_types_by_name['NanoPBOptions'] = _NANOPBOPTIONS
-
-class NanoPBOptions(message.Message):
-  __metaclass__ = reflection.GeneratedProtocolMessageType
-  DESCRIPTOR = _NANOPBOPTIONS
-  
-  # @@protoc_insertion_point(class_scope:NanoPBOptions)
-
-nanopb_fileopt.message_type = _NANOPBOPTIONS
-google.protobuf.descriptor_pb2.FileOptions.RegisterExtension(nanopb_fileopt)
-nanopb_msgopt.message_type = _NANOPBOPTIONS
-google.protobuf.descriptor_pb2.MessageOptions.RegisterExtension(nanopb_msgopt)
-nanopb_enumopt.message_type = _NANOPBOPTIONS
-google.protobuf.descriptor_pb2.EnumOptions.RegisterExtension(nanopb_enumopt)
-nanopb.message_type = _NANOPBOPTIONS
-google.protobuf.descriptor_pb2.FieldOptions.RegisterExtension(nanopb)
-# @@protoc_insertion_point(module_scope)

generator/plugin_pb2.py

@@ -1,161 +0,0 @@
-# Generated by the protocol buffer compiler.  DO NOT EDIT!
-
-from google.protobuf import descriptor
-from google.protobuf import message
-from google.protobuf import reflection
-from google.protobuf import descriptor_pb2
-# @@protoc_insertion_point(imports)
-
-
-import google.protobuf.descriptor_pb2
-
-DESCRIPTOR = descriptor.FileDescriptor(
-  name='plugin.proto',
-  package='google.protobuf.compiler',
-  serialized_pb='\n\x0cplugin.proto\x12\x18google.protobuf.compiler\x1a google/protobuf/descriptor.proto\"}\n\x14\x43odeGeneratorRequest\x12\x18\n\x10\x66ile_to_generate\x18\x01 \x03(\t\x12\x11\n\tparameter\x18\x02 \x01(\t\x12\x38\n\nproto_file\x18\x0f \x03(\x0b\x32$.google.protobuf.FileDescriptorProto\"\xaa\x01\n\x15\x43odeGeneratorResponse\x12\r\n\x05\x65rror\x18\x01 \x01(\t\x12\x42\n\x04\x66ile\x18\x0f \x03(\x0b\x32\x34.google.protobuf.compiler.CodeGeneratorResponse.File\x1a>\n\x04\x46ile\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x17\n\x0finsertion_point\x18\x02 \x01(\t\x12\x0f\n\x07\x63ontent\x18\x0f \x01(\t')
-
-
-
-
-_CODEGENERATORREQUEST = descriptor.Descriptor(
-  name='CodeGeneratorRequest',
-  full_name='google.protobuf.compiler.CodeGeneratorRequest',
-  filename=None,
-  file=DESCRIPTOR,
-  containing_type=None,
-  fields=[
-    descriptor.FieldDescriptor(
-      name='file_to_generate', full_name='google.protobuf.compiler.CodeGeneratorRequest.file_to_generate', index=0,
-      number=1, type=9, cpp_type=9, label=3,
-      has_default_value=False, default_value=[],
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='parameter', full_name='google.protobuf.compiler.CodeGeneratorRequest.parameter', index=1,
-      number=2, type=9, cpp_type=9, label=1,
-      has_default_value=False, default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='proto_file', full_name='google.protobuf.compiler.CodeGeneratorRequest.proto_file', index=2,
-      number=15, type=11, cpp_type=10, label=3,
-      has_default_value=False, default_value=[],
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-  ],
-  extensions=[
-  ],
-  nested_types=[],
-  enum_types=[
-  ],
-  options=None,
-  is_extendable=False,
-  extension_ranges=[],
-  serialized_start=76,
-  serialized_end=201,
-)
-
-
-_CODEGENERATORRESPONSE_FILE = descriptor.Descriptor(
-  name='File',
-  full_name='google.protobuf.compiler.CodeGeneratorResponse.File',
-  filename=None,
-  file=DESCRIPTOR,
-  containing_type=None,
-  fields=[
-    descriptor.FieldDescriptor(
-      name='name', full_name='google.protobuf.compiler.CodeGeneratorResponse.File.name', index=0,
-      number=1, type=9, cpp_type=9, label=1,
-      has_default_value=False, default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='insertion_point', full_name='google.protobuf.compiler.CodeGeneratorResponse.File.insertion_point', index=1,
-      number=2, type=9, cpp_type=9, label=1,
-      has_default_value=False, default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='content', full_name='google.protobuf.compiler.CodeGeneratorResponse.File.content', index=2,
-      number=15, type=9, cpp_type=9, label=1,
-      has_default_value=False, default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-  ],
-  extensions=[
-  ],
-  nested_types=[],
-  enum_types=[
-  ],
-  options=None,
-  is_extendable=False,
-  extension_ranges=[],
-  serialized_start=312,
-  serialized_end=374,
-)
-
-_CODEGENERATORRESPONSE = descriptor.Descriptor(
-  name='CodeGeneratorResponse',
-  full_name='google.protobuf.compiler.CodeGeneratorResponse',
-  filename=None,
-  file=DESCRIPTOR,
-  containing_type=None,
-  fields=[
-    descriptor.FieldDescriptor(
-      name='error', full_name='google.protobuf.compiler.CodeGeneratorResponse.error', index=0,
-      number=1, type=9, cpp_type=9, label=1,
-      has_default_value=False, default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='file', full_name='google.protobuf.compiler.CodeGeneratorResponse.file', index=1,
-      number=15, type=11, cpp_type=10, label=3,
-      has_default_value=False, default_value=[],
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-  ],
-  extensions=[
-  ],
-  nested_types=[_CODEGENERATORRESPONSE_FILE, ],
-  enum_types=[
-  ],
-  options=None,
-  is_extendable=False,
-  extension_ranges=[],
-  serialized_start=204,
-  serialized_end=374,
-)
-
-_CODEGENERATORREQUEST.fields_by_name['proto_file'].message_type = google.protobuf.descriptor_pb2._FILEDESCRIPTORPROTO
-_CODEGENERATORRESPONSE_FILE.containing_type = _CODEGENERATORRESPONSE;
-_CODEGENERATORRESPONSE.fields_by_name['file'].message_type = _CODEGENERATORRESPONSE_FILE
-DESCRIPTOR.message_types_by_name['CodeGeneratorRequest'] = _CODEGENERATORREQUEST
-DESCRIPTOR.message_types_by_name['CodeGeneratorResponse'] = _CODEGENERATORRESPONSE
-
-class CodeGeneratorRequest(message.Message):
-  __metaclass__ = reflection.GeneratedProtocolMessageType
-  DESCRIPTOR = _CODEGENERATORREQUEST
-  
-  # @@protoc_insertion_point(class_scope:google.protobuf.compiler.CodeGeneratorRequest)
-
-class CodeGeneratorResponse(message.Message):
-  __metaclass__ = reflection.GeneratedProtocolMessageType
-  
-  class File(message.Message):
-    __metaclass__ = reflection.GeneratedProtocolMessageType
-    DESCRIPTOR = _CODEGENERATORRESPONSE_FILE
-    
-    # @@protoc_insertion_point(class_scope:google.protobuf.compiler.CodeGeneratorResponse.File)
-  DESCRIPTOR = _CODEGENERATORRESPONSE
-  
-  # @@protoc_insertion_point(class_scope:google.protobuf.compiler.CodeGeneratorResponse)
-
-# @@protoc_insertion_point(module_scope)

generator/proto/Makefile

@@ -0,0 +1,4 @@
+all: nanopb_pb2.py plugin_pb2.py
+
+%_pb2.py: %.proto
+	protoc --python_out=. $<

generator/proto/google/protobuf/descriptor.proto

@@ -0,0 +1,620 @@
+// Protocol Buffers - Google's data interchange format
+// Copyright 2008 Google Inc.  All rights reserved.
+// http://code.google.com/p/protobuf/
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions are
+// met:
+//
+//     * Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+//     * Redistributions in binary form must reproduce the above
+// copyright notice, this list of conditions and the following disclaimer
+// in the documentation and/or other materials provided with the
+// distribution.
+//     * Neither the name of Google Inc. nor the names of its
+// contributors may be used to endorse or promote products derived from
+// this software without specific prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+// Author: kenton@google.com (Kenton Varda)
+//  Based on original Protocol Buffers design by
+//  Sanjay Ghemawat, Jeff Dean, and others.
+//
+// The messages in this file describe the definitions found in .proto files.
+// A valid .proto file can be translated directly to a FileDescriptorProto
+// without any other information (e.g. without reading its imports).
+
+
+
+package google.protobuf;
+option java_package = "com.google.protobuf";
+option java_outer_classname = "DescriptorProtos";
+
+// descriptor.proto must be optimized for speed because reflection-based
+// algorithms don't work during bootstrapping.
+option optimize_for = SPEED;
+
+// The protocol compiler can output a FileDescriptorSet containing the .proto
+// files it parses.
+message FileDescriptorSet {
+  repeated FileDescriptorProto file = 1;
+}
+
+// Describes a complete .proto file.
+message FileDescriptorProto {
+  optional string name = 1;       // file name, relative to root of source tree
+  optional string package = 2;    // e.g. "foo", "foo.bar", etc.
+
+  // Names of files imported by this file.
+  repeated string dependency = 3;
+  // Indexes of the public imported files in the dependency list above.
+  repeated int32 public_dependency = 10;
+  // Indexes of the weak imported files in the dependency list.
+  // For Google-internal migration only. Do not use.
+  repeated int32 weak_dependency = 11;
+
+  // All top-level definitions in this file.
+  repeated DescriptorProto message_type = 4;
+  repeated EnumDescriptorProto enum_type = 5;
+  repeated ServiceDescriptorProto service = 6;
+  repeated FieldDescriptorProto extension = 7;
+
+  optional FileOptions options = 8;
+
+  // This field contains optional information about the original source code.
+  // You may safely remove this entire field whithout harming runtime
+  // functionality of the descriptors -- the information is needed only by
+  // development tools.
+  optional SourceCodeInfo source_code_info = 9;
+}
+
+// Describes a message type.
+message DescriptorProto {
+  optional string name = 1;
+
+  repeated FieldDescriptorProto field = 2;
+  repeated FieldDescriptorProto extension = 6;
+
+  repeated DescriptorProto nested_type = 3;
+  repeated EnumDescriptorProto enum_type = 4;
+
+  message ExtensionRange {
+    optional int32 start = 1;
+    optional int32 end = 2;
+  }
+  repeated ExtensionRange extension_range = 5;
+
+  optional MessageOptions options = 7;
+}
+
+// Describes a field within a message.
+message FieldDescriptorProto {
+  enum Type {
+    // 0 is reserved for errors.
+    // Order is weird for historical reasons.
+    TYPE_DOUBLE         = 1;
+    TYPE_FLOAT          = 2;
+    // Not ZigZag encoded.  Negative numbers take 10 bytes.  Use TYPE_SINT64 if
+    // negative values are likely.
+    TYPE_INT64          = 3;
+    TYPE_UINT64         = 4;
+    // Not ZigZag encoded.  Negative numbers take 10 bytes.  Use TYPE_SINT32 if
+    // negative values are likely.
+    TYPE_INT32          = 5;
+    TYPE_FIXED64        = 6;
+    TYPE_FIXED32        = 7;
+    TYPE_BOOL           = 8;
+    TYPE_STRING         = 9;
+    TYPE_GROUP          = 10;  // Tag-delimited aggregate.
+    TYPE_MESSAGE        = 11;  // Length-delimited aggregate.
+
+    // New in version 2.
+    TYPE_BYTES          = 12;
+    TYPE_UINT32         = 13;
+    TYPE_ENUM           = 14;
+    TYPE_SFIXED32       = 15;
+    TYPE_SFIXED64       = 16;
+    TYPE_SINT32         = 17;  // Uses ZigZag encoding.
+    TYPE_SINT64         = 18;  // Uses ZigZag encoding.
+  };
+
+  enum Label {
+    // 0 is reserved for errors
+    LABEL_OPTIONAL      = 1;
+    LABEL_REQUIRED      = 2;
+    LABEL_REPEATED      = 3;
+    // TODO(sanjay): Should we add LABEL_MAP?
+  };
+
+  optional string name = 1;
+  optional int32 number = 3;
+  optional Label label = 4;
+
+  // If type_name is set, this need not be set.  If both this and type_name
+  // are set, this must be either TYPE_ENUM or TYPE_MESSAGE.
+  optional Type type = 5;
+
+  // For message and enum types, this is the name of the type.  If the name
+  // starts with a '.', it is fully-qualified.  Otherwise, C++-like scoping
+  // rules are used to find the type (i.e. first the nested types within this
+  // message are searched, then within the parent, on up to the root
+  // namespace).
+  optional string type_name = 6;
+
+  // For extensions, this is the name of the type being extended.  It is
+  // resolved in the same manner as type_name.
+  optional string extendee = 2;
+
+  // For numeric types, contains the original text representation of the value.
+  // For booleans, "true" or "false".
+  // For strings, contains the default text contents (not escaped in any way).
+  // For bytes, contains the C escaped value.  All bytes >= 128 are escaped.
+  // TODO(kenton):  Base-64 encode?
+  optional string default_value = 7;
+
+  optional FieldOptions options = 8;
+}
+
+// Describes an enum type.
+message EnumDescriptorProto {
+  optional string name = 1;
+
+  repeated EnumValueDescriptorProto value = 2;
+
+  optional EnumOptions options = 3;
+}
+
+// Describes a value within an enum.
+message EnumValueDescriptorProto {
+  optional string name = 1;
+  optional int32 number = 2;
+
+  optional EnumValueOptions options = 3;
+}
+
+// Describes a service.
+message ServiceDescriptorProto {
+  optional string name = 1;
+  repeated MethodDescriptorProto method = 2;
+
+  optional ServiceOptions options = 3;
+}
+
+// Describes a method of a service.
+message MethodDescriptorProto {
+  optional string name = 1;
+
+  // Input and output type names.  These are resolved in the same way as
+  // FieldDescriptorProto.type_name, but must refer to a message type.
+  optional string input_type = 2;
+  optional string output_type = 3;
+
+  optional MethodOptions options = 4;
+}
+
+
+// ===================================================================
+// Options
+
+// Each of the definitions above may have "options" attached.  These are
+// just annotations which may cause code to be generated slightly differently
+// or may contain hints for code that manipulates protocol messages.
+//
+// Clients may define custom options as extensions of the *Options messages.
+// These extensions may not yet be known at parsing time, so the parser cannot
+// store the values in them.  Instead it stores them in a field in the *Options
+// message called uninterpreted_option. This field must have the same name
+// across all *Options messages. We then use this field to populate the
+// extensions when we build a descriptor, at which point all protos have been
+// parsed and so all extensions are known.
+//
+// Extension numbers for custom options may be chosen as follows:
+// * For options which will only be used within a single application or
+//   organization, or for experimental options, use field numbers 50000
+//   through 99999.  It is up to you to ensure that you do not use the
+//   same number for multiple options.
+// * For options which will be published and used publicly by multiple
+//   independent entities, e-mail protobuf-global-extension-registry@google.com
+//   to reserve extension numbers. Simply provide your project name (e.g.
+//   Object-C plugin) and your porject website (if available) -- there's no need
+//   to explain how you intend to use them. Usually you only need one extension
+//   number. You can declare multiple options with only one extension number by
+//   putting them in a sub-message. See the Custom Options section of the docs
+//   for examples:
+//   http://code.google.com/apis/protocolbuffers/docs/proto.html#options
+//   If this turns out to be popular, a web service will be set up
+//   to automatically assign option numbers.
+
+
+message FileOptions {
+
+  // Sets the Java package where classes generated from this .proto will be
+  // placed.  By default, the proto package is used, but this is often
+  // inappropriate because proto packages do not normally start with backwards
+  // domain names.
+  optional string java_package = 1;
+
+
+  // If set, all the classes from the .proto file are wrapped in a single
+  // outer class with the given name.  This applies to both Proto1
+  // (equivalent to the old "--one_java_file" option) and Proto2 (where
+  // a .proto always translates to a single class, but you may want to
+  // explicitly choose the class name).
+  optional string java_outer_classname = 8;
+
+  // If set true, then the Java code generator will generate a separate .java
+  // file for each top-level message, enum, and service defined in the .proto
+  // file.  Thus, these types will *not* be nested inside the outer class
+  // named by java_outer_classname.  However, the outer class will still be
+  // generated to contain the file's getDescriptor() method as well as any
+  // top-level extensions defined in the file.
+  optional bool java_multiple_files = 10 [default=false];
+
+  // If set true, then the Java code generator will generate equals() and
+  // hashCode() methods for all messages defined in the .proto file. This is
+  // purely a speed optimization, as the AbstractMessage base class includes
+  // reflection-based implementations of these methods.
+  optional bool java_generate_equals_and_hash = 20 [default=false];
+
+  // Generated classes can be optimized for speed or code size.
+  enum OptimizeMode {
+    SPEED = 1;        // Generate complete code for parsing, serialization,
+                      // etc.
+    CODE_SIZE = 2;    // Use ReflectionOps to implement these methods.
+    LITE_RUNTIME = 3; // Generate code using MessageLite and the lite runtime.
+  }
+  optional OptimizeMode optimize_for = 9 [default=SPEED];
+
+  // Sets the Go package where structs generated from this .proto will be
+  // placed.  There is no default.
+  optional string go_package = 11;
+
+
+
+  // Should generic services be generated in each language?  "Generic" services
+  // are not specific to any particular RPC system.  They are generated by the
+  // main code generators in each language (without additional plugins).
+  // Generic services were the only kind of service generation supported by
+  // early versions of proto2.
+  //
+  // Generic services are now considered deprecated in favor of using plugins
+  // that generate code specific to your particular RPC system.  Therefore,
+  // these default to false.  Old code which depends on generic services should
+  // explicitly set them to true.
+  optional bool cc_generic_services = 16 [default=false];
+  optional bool java_generic_services = 17 [default=false];
+  optional bool py_generic_services = 18 [default=false];
+
+  // The parser stores options it doesn't recognize here. See above.
+  repeated UninterpretedOption uninterpreted_option = 999;
+
+  // Clients can define custom options in extensions of this message. See above.
+  extensions 1000 to max;
+}
+
+message MessageOptions {
+  // Set true to use the old proto1 MessageSet wire format for extensions.
+  // This is provided for backwards-compatibility with the MessageSet wire
+  // format.  You should not use this for any other reason:  It's less
+  // efficient, has fewer features, and is more complicated.
+  //
+  // The message must be defined exactly as follows:
+  //   message Foo {
+  //     option message_set_wire_format = true;
+  //     extensions 4 to max;
+  //   }
+  // Note that the message cannot have any defined fields; MessageSets only
+  // have extensions.
+  //
+  // All extensions of your type must be singular messages; e.g. they cannot
+  // be int32s, enums, or repeated messages.
+  //
+  // Because this is an option, the above two restrictions are not enforced by
+  // the protocol compiler.
+  optional bool message_set_wire_format = 1 [default=false];
+
+  // Disables the generation of the standard "descriptor()" accessor, which can
+  // conflict with a field of the same name.  This is meant to make migration
+  // from proto1 easier; new code should avoid fields named "descriptor".
+  optional bool no_standard_descriptor_accessor = 2 [default=false];
+
+  // The parser stores options it doesn't recognize here. See above.
+  repeated UninterpretedOption uninterpreted_option = 999;
+
+  // Clients can define custom options in extensions of this message. See above.
+  extensions 1000 to max;
+}
+
+message FieldOptions {
+  // The ctype option instructs the C++ code generator to use a different
+  // representation of the field than it normally would.  See the specific
+  // options below.  This option is not yet implemented in the open source
+  // release -- sorry, we'll try to include it in a future version!
+  optional CType ctype = 1 [default = STRING];
+  enum CType {
+    // Default mode.
+    STRING = 0;
+
+    CORD = 1;
+
+    STRING_PIECE = 2;
+  }
+  // The packed option can be enabled for repeated primitive fields to enable
+  // a more efficient representation on the wire. Rather than repeatedly
+  // writing the tag and type for each element, the entire array is encoded as
+  // a single length-delimited blob.
+  optional bool packed = 2;
+
+
+
+  // Should this field be parsed lazily?  Lazy applies only to message-type
+  // fields.  It means that when the outer message is initially parsed, the
+  // inner message's contents will not be parsed but instead stored in encoded
+  // form.  The inner message will actually be parsed when it is first accessed.
+  //
+  // This is only a hint.  Implementations are free to choose whether to use
+  // eager or lazy parsing regardless of the value of this option.  However,
+  // setting this option true suggests that the protocol author believes that
+  // using lazy parsing on this field is worth the additional bookkeeping
+  // overhead typically needed to implement it.
+  //
+  // This option does not affect the public interface of any generated code;
+  // all method signatures remain the same.  Furthermore, thread-safety of the
+  // interface is not affected by this option; const methods remain safe to
+  // call from multiple threads concurrently, while non-const methods continue
+  // to require exclusive access.
+  //
+  //
+  // Note that implementations may choose not to check required fields within
+  // a lazy sub-message.  That is, calling IsInitialized() on the outher message
+  // may return true even if the inner message has missing required fields.
+  // This is necessary because otherwise the inner message would have to be
+  // parsed in order to perform the check, defeating the purpose of lazy
+  // parsing.  An implementation which chooses not to check required fields
+  // must be consistent about it.  That is, for any particular sub-message, the
+  // implementation must either *always* check its required fields, or *never*
+  // check its required fields, regardless of whether or not the message has
+  // been parsed.
+  optional bool lazy = 5 [default=false];
+
+  // Is this field deprecated?
+  // Depending on the target platform, this can emit Deprecated annotations
+  // for accessors, or it will be completely ignored; in the very least, this
+  // is a formalization for deprecating fields.
+  optional bool deprecated = 3 [default=false];
+
+  // EXPERIMENTAL.  DO NOT USE.
+  // For "map" fields, the name of the field in the enclosed type that
+  // is the key for this map.  For example, suppose we have:
+  //   message Item {
+  //     required string name = 1;
+  //     required string value = 2;
+  //   }
+  //   message Config {
+  //     repeated Item items = 1 [experimental_map_key="name"];
+  //   }
+  // In this situation, the map key for Item will be set to "name".
+  // TODO: Fully-implement this, then remove the "experimental_" prefix.
+  optional string experimental_map_key = 9;
+
+  // For Google-internal migration only. Do not use.
+  optional bool weak = 10 [default=false];
+
+  // The parser stores options it doesn't recognize here. See above.
+  repeated UninterpretedOption uninterpreted_option = 999;
+
+  // Clients can define custom options in extensions of this message. See above.
+  extensions 1000 to max;
+}
+
+message EnumOptions {
+
+  // Set this option to false to disallow mapping different tag names to a same
+  // value.
+  optional bool allow_alias = 2 [default=true];
+
+  // The parser stores options it doesn't recognize here. See above.
+  repeated UninterpretedOption uninterpreted_option = 999;
+
+  // Clients can define custom options in extensions of this message. See above.
+  extensions 1000 to max;
+}
+
+message EnumValueOptions {
+  // The parser stores options it doesn't recognize here. See above.
+  repeated UninterpretedOption uninterpreted_option = 999;
+
+  // Clients can define custom options in extensions of this message. See above.
+  extensions 1000 to max;
+}
+
+message ServiceOptions {
+
+  // Note:  Field numbers 1 through 32 are reserved for Google's internal RPC
+  //   framework.  We apologize for hoarding these numbers to ourselves, but
+  //   we were already using them long before we decided to release Protocol
+  //   Buffers.
+
+  // The parser stores options it doesn't recognize here. See above.
+  repeated UninterpretedOption uninterpreted_option = 999;
+
+  // Clients can define custom options in extensions of this message. See above.
+  extensions 1000 to max;
+}
+
+message MethodOptions {
+
+  // Note:  Field numbers 1 through 32 are reserved for Google's internal RPC
+  //   framework.  We apologize for hoarding these numbers to ourselves, but
+  //   we were already using them long before we decided to release Protocol
+  //   Buffers.
+
+  // The parser stores options it doesn't recognize here. See above.
+  repeated UninterpretedOption uninterpreted_option = 999;
+
+  // Clients can define custom options in extensions of this message. See above.
+  extensions 1000 to max;
+}
+
+
+// A message representing a option the parser does not recognize. This only
+// appears in options protos created by the compiler::Parser class.
+// DescriptorPool resolves these when building Descriptor objects. Therefore,
+// options protos in descriptor objects (e.g. returned by Descriptor::options(),
+// or produced by Descriptor::CopyTo()) will never have UninterpretedOptions
+// in them.
+message UninterpretedOption {
+  // The name of the uninterpreted option.  Each string represents a segment in
+  // a dot-separated name.  is_extension is true iff a segment represents an
+  // extension (denoted with parentheses in options specs in .proto files).
+  // E.g.,{ ["foo", false], ["bar.baz", true], ["qux", false] } represents
+  // "foo.(bar.baz).qux".
+  message NamePart {
+    required string name_part = 1;
+    required bool is_extension = 2;
+  }
+  repeated NamePart name = 2;
+
+  // The value of the uninterpreted option, in whatever type the tokenizer
+  // identified it as during parsing. Exactly one of these should be set.
+  optional string identifier_value = 3;
+  optional uint64 positive_int_value = 4;
+  optional int64 negative_int_value = 5;
+  optional double double_value = 6;
+  optional bytes string_value = 7;
+  optional string aggregate_value = 8;
+}
+
+// ===================================================================
+// Optional source code info
+
+// Encapsulates information about the original source file from which a
+// FileDescriptorProto was generated.
+message SourceCodeInfo {
+  // A Location identifies a piece of source code in a .proto file which
+  // corresponds to a particular definition.  This information is intended
+  // to be useful to IDEs, code indexers, documentation generators, and similar
+  // tools.
+  //
+  // For example, say we have a file like:
+  //   message Foo {
+  //     optional string foo = 1;
+  //   }
+  // Let's look at just the field definition:
+  //   optional string foo = 1;
+  //   ^       ^^     ^^  ^  ^^^
+  //   a       bc     de  f  ghi
+  // We have the following locations:
+  //   span   path               represents
+  //   [a,i)  [ 4, 0, 2, 0 ]     The whole field definition.
+  //   [a,b)  [ 4, 0, 2, 0, 4 ]  The label (optional).
+  //   [c,d)  [ 4, 0, 2, 0, 5 ]  The type (string).
+  //   [e,f)  [ 4, 0, 2, 0, 1 ]  The name (foo).
+  //   [g,h)  [ 4, 0, 2, 0, 3 ]  The number (1).
+  //
+  // Notes:
+  // - A location may refer to a repeated field itself (i.e. not to any
+  //   particular index within it).  This is used whenever a set of elements are
+  //   logically enclosed in a single code segment.  For example, an entire
+  //   extend block (possibly containing multiple extension definitions) will
+  //   have an outer location whose path refers to the "extensions" repeated
+  //   field without an index.
+  // - Multiple locations may have the same path.  This happens when a single
+  //   logical declaration is spread out across multiple places.  The most
+  //   obvious example is the "extend" block again -- there may be multiple
+  //   extend blocks in the same scope, each of which will have the same path.
+  // - A location's span is not always a subset of its parent's span.  For
+  //   example, the "extendee" of an extension declaration appears at the
+  //   beginning of the "extend" block and is shared by all extensions within
+  //   the block.
+  // - Just because a location's span is a subset of some other location's span
+  //   does not mean that it is a descendent.  For example, a "group" defines
+  //   both a type and a field in a single declaration.  Thus, the locations
+  //   corresponding to the type and field and their components will overlap.
+  // - Code which tries to interpret locations should probably be designed to
+  //   ignore those that it doesn't understand, as more types of locations could
+  //   be recorded in the future.
+  repeated Location location = 1;
+  message Location {
+    // Identifies which part of the FileDescriptorProto was defined at this
+    // location.
+    //
+    // Each element is a field number or an index.  They form a path from
+    // the root FileDescriptorProto to the place where the definition.  For
+    // example, this path:
+    //   [ 4, 3, 2, 7, 1 ]
+    // refers to:
+    //   file.message_type(3)  // 4, 3
+    //       .field(7)         // 2, 7
+    //       .name()           // 1
+    // This is because FileDescriptorProto.message_type has field number 4:
+    //   repeated DescriptorProto message_type = 4;
+    // and DescriptorProto.field has field number 2:
+    //   repeated FieldDescriptorProto field = 2;
+    // and FieldDescriptorProto.name has field number 1:
+    //   optional string name = 1;
+    //
+    // Thus, the above path gives the location of a field name.  If we removed
+    // the last element:
+    //   [ 4, 3, 2, 7 ]
+    // this path refers to the whole field declaration (from the beginning
+    // of the label to the terminating semicolon).
+    repeated int32 path = 1 [packed=true];
+
+    // Always has exactly three or four elements: start line, start column,
+    // end line (optional, otherwise assumed same as start line), end column.
+    // These are packed into a single field for efficiency.  Note that line
+    // and column numbers are zero-based -- typically you will want to add
+    // 1 to each before displaying to a user.
+    repeated int32 span = 2 [packed=true];
+
+    // If this SourceCodeInfo represents a complete declaration, these are any
+    // comments appearing before and after the declaration which appear to be
+    // attached to the declaration.
+    //
+    // A series of line comments appearing on consecutive lines, with no other
+    // tokens appearing on those lines, will be treated as a single comment.
+    //
+    // Only the comment content is provided; comment markers (e.g. //) are
+    // stripped out.  For block comments, leading whitespace and an asterisk
+    // will be stripped from the beginning of each line other than the first.
+    // Newlines are included in the output.
+    //
+    // Examples:
+    //
+    //   optional int32 foo = 1;  // Comment attached to foo.
+    //   // Comment attached to bar.
+    //   optional int32 bar = 2;
+    //
+    //   optional string baz = 3;
+    //   // Comment attached to baz.
+    //   // Another line attached to baz.
+    //
+    //   // Comment attached to qux.
+    //   //
+    //   // Another line attached to qux.
+    //   optional double qux = 4;
+    //
+    //   optional string corge = 5;
+    //   /* Block comment attached
+    //    * to corge.  Leading asterisks
+    //    * will be removed. */
+    //   /* Block comment attached to
+    //    * grault. */
+    //   optional int32 grault = 6;
+    optional string leading_comments = 3;
+    optional string trailing_comments = 4;
+  }
+}

generator/proto/nanopb.proto

@@ -12,6 +12,7 @@ option java_package = "fi.kapsi.koti.jpa.nanopb";
 enum FieldType {
     FT_DEFAULT = 0; // Automatically decide field type, generate static field if possible.
     FT_CALLBACK = 1; // Always generate a callback field.
+    FT_POINTER = 4; // Always generate a dynamically allocated field.
     FT_STATIC = 2; // Generate a static field or raise an exception if not possible.
     FT_IGNORE = 3; // Ignore the field completely.
 }
@@ -36,6 +37,9 @@ message NanoPBOptions {
   // Note: this cannot be used on CPUs that break on unaligned
   // accesses to variables.
   optional bool packed_struct = 5 [default = false];
+  
+  // Skip this message
+  optional bool skip_message = 6 [default = false];
 }
 
 // Extensions to protoc 'Descriptor' type in order to define options

generator/protoc-gen-nanopb

@@ -0,0 +1,13 @@
+#!/bin/sh
+
+# This file is used to invoke nanopb_generator.py as a plugin
+# to protoc on Linux and other *nix-style systems.
+# Use it like this:
+# protoc --plugin=nanopb=..../protoc-gen-nanopb --nanopb_out=dir foo.proto
+#
+# Note that if you use the binary package of nanopb, the protoc
+# path is already set up properly and there is no need to give
+# --plugin= on the command line.
+
+MYPATH=$(dirname "$0")
+exec python "$MYPATH/nanopb_generator.py" --protoc-plugin

generator/protoc-gen-nanopb.bat

@@ -0,0 +1,12 @@
+@echo off
+:: This file is used to invoke nanopb_generator.py as a plugin
+:: to protoc on Windows.
+:: Use it like this:
+:: protoc --plugin=nanopb=..../protoc-gen-nanopb.bat --nanopb_out=dir foo.proto
+::
+:: Note that if you use the binary package of nanopb, the protoc
+:: path is already set up properly and there is no need to give
+:: --plugin= on the command line.
+
+set mydir=%~dp0
+python "%mydir%\nanopb_generator.py" --protoc-plugin

pb.h

@@ -2,14 +2,17 @@
  * stuff. For the high-level interface, see pb_encode.h and pb_decode.h.
  */
 
-#ifndef _PB_H_
-#define _PB_H_
+#ifndef PB_H_INCLUDED
+#define PB_H_INCLUDED
 
 /*****************************************************************
  * Nanopb compilation time options. You can change these here by *
  * uncommenting the lines, or on the compiler command line.      *
  *****************************************************************/
 
+/* Enable support for dynamically allocated fields */
+/* #define PB_ENABLE_MALLOC 1 */
+
 /* Define this if your CPU architecture is big endian, i.e. it
  * stores the most-significant byte first. */
 /* #define __BIG_ENDIAN__ 1 */
@@ -43,7 +46,7 @@
 
 /* Version of the nanopb library. Just in case you want to check it in
  * your own program. */
-#define NANOPB_VERSION nanopb-0.2.3-dev
+#define NANOPB_VERSION nanopb-0.3.1
 
 /* Include all the system headers needed by nanopb. You will need the
  * definitions of the following:
@@ -63,6 +66,10 @@
 #include <stddef.h>
 #include <stdbool.h>
 #include <string.h>
+
+#ifdef PB_ENABLE_MALLOC
+#include <stdlib.h>
+#endif
 #endif
 
 /* Macro for defining packed structures (compiler dependent).
@@ -73,8 +80,8 @@
 #   define PB_PACKED_STRUCT_START
 #   define PB_PACKED_STRUCT_END
 #   define pb_packed __attribute__((packed))
-#elif defined(__ICCARM__)
-    /* For IAR ARM compiler */
+#elif defined(__ICCARM__) || defined(__CC_ARM)
+    /* For IAR ARM and Keil MDK-ARM compilers */
 #   define PB_PACKED_STRUCT_START _Pragma("pack(push, 1)")
 #   define PB_PACKED_STRUCT_END _Pragma("pack(pop)")
 #   define pb_packed
@@ -91,17 +98,27 @@
 #endif
 
 /* Handly macro for suppressing unreferenced-parameter compiler warnings. */
-#ifndef UNUSED
-#define UNUSED(x) (void)(x)
+#ifndef PB_UNUSED
+#define PB_UNUSED(x) (void)(x)
 #endif
 
 /* Compile-time assertion, used for checking compatible compilation options.
- * If this fails on your compiler for some reason, use #define STATIC_ASSERT
- * to disable it. */
-#ifndef STATIC_ASSERT
-#define STATIC_ASSERT(COND,MSG) typedef char STATIC_ASSERT_MSG(MSG, __LINE__, __COUNTER__)[(COND)?1:-1];
-#define STATIC_ASSERT_MSG(MSG, LINE, COUNTER) STATIC_ASSERT_MSG_(MSG, LINE, COUNTER)
-#define STATIC_ASSERT_MSG_(MSG, LINE, COUNTER) static_assertion_##MSG##LINE##COUNTER
+ * If this does not work properly on your compiler, use
+ * #define PB_NO_STATIC_ASSERT to disable it.
+ *
+ * But before doing that, check carefully the error message / place where it
+ * comes from to see if the error has a real cause. Unfortunately the error
+ * message is not always very clear to read, but you can see the reason better
+ * in the place where the PB_STATIC_ASSERT macro was called.
+ */
+#ifndef PB_NO_STATIC_ASSERT
+#ifndef PB_STATIC_ASSERT
+#define PB_STATIC_ASSERT(COND,MSG) typedef char PB_STATIC_ASSERT_MSG(MSG, __LINE__, __COUNTER__)[(COND)?1:-1];
+#define PB_STATIC_ASSERT_MSG(MSG, LINE, COUNTER) PB_STATIC_ASSERT_MSG_(MSG, LINE, COUNTER)
+#define PB_STATIC_ASSERT_MSG_(MSG, LINE, COUNTER) pb_static_assertion_##MSG##LINE##COUNTER
+#endif
+#else
+#define PB_STATIC_ASSERT(COND,MSG)
 #endif
 
 /* Number of required fields to keep track of. */
@@ -116,11 +133,6 @@
 /* List of possible field types. These are used in the autogenerated code.
  * Least-significant 4 bits tell the scalar type
  * Most-significant 4 bits specify repeated/required/packed etc.
- * 
- * INT32 and UINT32 are treated the same, as are (U)INT64 and (S)FIXED*
- * These types are simply casted to correct field type when they are
- * assigned to the memory pointer.
- * SINT* is different, though, because it is zig-zag coded.
  */
 
 typedef uint8_t pb_type_t;
@@ -128,32 +140,33 @@ typedef uint8_t pb_type_t;
 /**** Field data types ****/
 
 /* Numeric types */
-#define PB_LTYPE_VARINT 0x00 /* int32, uint32, int64, uint64, bool, enum */
-#define PB_LTYPE_SVARINT 0x01 /* sint32, sint64 */
-#define PB_LTYPE_FIXED32 0x02 /* fixed32, sfixed32, float */
-#define PB_LTYPE_FIXED64 0x03 /* fixed64, sfixed64, double */
+#define PB_LTYPE_VARINT  0x00 /* int32, int64, enum, bool */
+#define PB_LTYPE_UVARINT 0x01 /* uint32, uint64 */
+#define PB_LTYPE_SVARINT 0x02 /* sint32, sint64 */
+#define PB_LTYPE_FIXED32 0x03 /* fixed32, sfixed32, float */
+#define PB_LTYPE_FIXED64 0x04 /* fixed64, sfixed64, double */
 
 /* Marker for last packable field type. */
-#define PB_LTYPE_LAST_PACKABLE 0x03
+#define PB_LTYPE_LAST_PACKABLE 0x04
 
 /* Byte array with pre-allocated buffer.
  * data_size is the length of the allocated PB_BYTES_ARRAY structure. */
-#define PB_LTYPE_BYTES 0x04
+#define PB_LTYPE_BYTES 0x05
 
 /* String with pre-allocated buffer.
  * data_size is the maximum length. */
-#define PB_LTYPE_STRING 0x05
+#define PB_LTYPE_STRING 0x06
 
 /* Submessage
  * submsg_fields is pointer to field descriptions */
-#define PB_LTYPE_SUBMESSAGE 0x06
+#define PB_LTYPE_SUBMESSAGE 0x07
 
 /* Extension pseudo-field
  * The field contains a pointer to pb_extension_t */
-#define PB_LTYPE_EXTENSION 0x07
+#define PB_LTYPE_EXTENSION 0x08
 
 /* Number of declared LTYPES */
-#define PB_LTYPES_COUNT 8
+#define PB_LTYPES_COUNT 9
 #define PB_LTYPE_MASK 0x0F
 
 /**** Field repetition rules ****/
@@ -166,6 +179,7 @@ typedef uint8_t pb_type_t;
 /**** Field allocation types ****/
  
 #define PB_ATYPE_STATIC   0x00
+#define PB_ATYPE_POINTER  0x80
 #define PB_ATYPE_CALLBACK 0x40
 #define PB_ATYPE_MASK     0xC0
 
@@ -173,6 +187,23 @@ typedef uint8_t pb_type_t;
 #define PB_HTYPE(x) ((x) & PB_HTYPE_MASK)
 #define PB_LTYPE(x) ((x) & PB_LTYPE_MASK)
 
+/* Data type used for storing sizes of struct fields
+ * and array counts.
+ */
+#if defined(PB_FIELD_32BIT)
+#define PB_SIZE_MAX ((uint32_t)-1)
+    typedef uint32_t pb_size_t;
+    typedef int32_t pb_ssize_t;
+#elif defined(PB_FIELD_16BIT)
+#define PB_SIZE_MAX ((uint16_t)-1)
+    typedef uint16_t pb_size_t;
+    typedef int16_t pb_ssize_t;
+#else
+#define PB_SIZE_MAX ((uint8_t)-1)
+    typedef uint8_t pb_size_t;
+    typedef int8_t pb_ssize_t;
+#endif
+
 /* This structure is used in auto-generated constants
  * to specify struct fields.
  * You can change field sizes if you need structures
@@ -182,31 +213,14 @@ typedef uint8_t pb_type_t;
  * PB_FIELD_32BIT.
  */
 PB_PACKED_STRUCT_START
-typedef struct _pb_field_t pb_field_t;
-struct _pb_field_t {
-
-#if !defined(PB_FIELD_16BIT) && !defined(PB_FIELD_32BIT)
-    uint8_t tag;
+typedef struct pb_field_s pb_field_t;
+struct pb_field_s {
+    pb_size_t tag;
     pb_type_t type;
-    uint8_t data_offset; /* Offset of field data, relative to previous field. */
-    int8_t size_offset; /* Offset of array size or has-boolean, relative to data */
-    uint8_t data_size; /* Data size in bytes for a single item */
-    uint8_t array_size; /* Maximum number of entries in array */
-#elif defined(PB_FIELD_16BIT) && !defined(PB_FIELD_32BIT)
-    uint16_t tag;
-    pb_type_t type;
-    uint8_t data_offset;
-    int8_t size_offset;
-    uint16_t data_size;
-    uint16_t array_size;
-#else
-    uint32_t tag;
-    pb_type_t type;
-    uint8_t data_offset;
-    int8_t size_offset;
-    uint32_t data_size;
-    uint32_t array_size;
-#endif
+    pb_size_t data_offset; /* Offset of field data, relative to previous field. */
+    pb_ssize_t size_offset; /* Offset of array size or has-boolean, relative to data */
+    pb_size_t data_size; /* Data size in bytes for a single item */
+    pb_size_t array_size; /* Maximum number of entries in array */
     
     /* Field definitions for submessage
      * OR default value for all other non-array, non-callback types
@@ -215,16 +229,33 @@ struct _pb_field_t {
 } pb_packed;
 PB_PACKED_STRUCT_END
 
+/* Make sure that the standard integer types are of the expected sizes.
+ * All kinds of things may break otherwise.. atleast all fixed* types.
+ *
+ * If you get errors here, it probably means that your stdint.h is not
+ * correct for your platform.
+ */
+PB_STATIC_ASSERT(sizeof(int8_t) == 1, INT8_T_WRONG_SIZE)
+PB_STATIC_ASSERT(sizeof(uint8_t) == 1, UINT8_T_WRONG_SIZE)
+PB_STATIC_ASSERT(sizeof(int16_t) == 2, INT16_T_WRONG_SIZE)
+PB_STATIC_ASSERT(sizeof(uint16_t) == 2, UINT16_T_WRONG_SIZE)
+PB_STATIC_ASSERT(sizeof(int32_t) == 4, INT32_T_WRONG_SIZE)
+PB_STATIC_ASSERT(sizeof(uint32_t) == 4, UINT32_T_WRONG_SIZE)
+PB_STATIC_ASSERT(sizeof(int64_t) == 8, INT64_T_WRONG_SIZE)
+PB_STATIC_ASSERT(sizeof(uint64_t) == 8, UINT64_T_WRONG_SIZE)
+
 /* This structure is used for 'bytes' arrays.
  * It has the number of bytes in the beginning, and after that an array.
  * Note that actual structs used will have a different length of bytes array.
  */
-struct _pb_bytes_array_t {
-    size_t size;
+#define PB_BYTES_ARRAY_T(n) struct { pb_size_t size; uint8_t bytes[n]; }
+#define PB_BYTES_ARRAY_T_ALLOCSIZE(n) ((size_t)n + offsetof(pb_bytes_array_t, bytes))
+
+struct pb_bytes_array_s {
+    pb_size_t size;
     uint8_t bytes[1];
 };
-
-typedef struct _pb_bytes_array_t pb_bytes_array_t;
+typedef struct pb_bytes_array_s pb_bytes_array_t;
 
 /* This structure is used for giving the callback function.
  * It is stored in the message structure and filled in by the method that
@@ -244,10 +275,10 @@ typedef struct _pb_bytes_array_t pb_bytes_array_t;
  *
  * The callback can be null if you want to skip a field.
  */
-typedef struct _pb_istream_t pb_istream_t;
-typedef struct _pb_ostream_t pb_ostream_t;
-typedef struct _pb_callback_t pb_callback_t;
-struct _pb_callback_t {
+typedef struct pb_istream_s pb_istream_t;
+typedef struct pb_ostream_s pb_ostream_t;
+typedef struct pb_callback_s pb_callback_t;
+struct pb_callback_s {
 #ifdef PB_OLD_CALLBACK_STYLE
     /* Deprecated since nanopb-0.2.1 */
     union {
@@ -280,9 +311,9 @@ typedef enum {
  * if you want to catch all unknown fields, you can also create a custom
  * pb_extension_type_t with your own callback.
  */
-typedef struct _pb_extension_type_t pb_extension_type_t;
-typedef struct _pb_extension_t pb_extension_t;
-struct _pb_extension_type_t {
+typedef struct pb_extension_type_s pb_extension_type_t;
+typedef struct pb_extension_s pb_extension_t;
+struct pb_extension_type_s {
     /* Called for each unknown field in the message.
      * If you handle the field, read off all of its data and return true.
      * If you do not handle the field, do not read anything and return true.
@@ -304,7 +335,7 @@ struct _pb_extension_type_t {
     const void *arg;
 };
 
-struct _pb_extension_t {
+struct pb_extension_s {
     /* Type describing the extension field. Usually you'll initialize
      * this to a pointer to the automatically generated structure. */
     const pb_extension_type_t *type;
@@ -317,8 +348,26 @@ struct _pb_extension_t {
      * If this extension does not match a field, the next handler is
      * automatically called. */
     pb_extension_t *next;
+
+    /* The decoder sets this to true if the extension was found.
+     * Ignored for encoding. */
+    bool found;
 };
 
+/* Memory allocation functions to use. You can define pb_realloc and
+ * pb_free to custom functions if you want. */
+#ifdef PB_ENABLE_MALLOC
+#   ifndef pb_realloc
+#       define pb_realloc(ptr, size) realloc(ptr, size)
+#   endif
+#   ifndef pb_free
+#       define pb_free(ptr) free(ptr)
+#   endif
+#endif
+
+/* This is used to inform about need to regenerate .pb.h/.pb.c files. */
+#define PB_PROTO_HEADER_VERSION 30
+
 /* These macros are used to declare pb_field_t's in the constant array. */
 /* Size of a structure member, in bytes. */
 #define pb_membersize(st, m) (sizeof ((st*)0)->m)
@@ -326,16 +375,19 @@ struct _pb_extension_t {
 #define pb_arraysize(st, m) (pb_membersize(st, m) / pb_membersize(st, m[0]))
 /* Delta from start of one member to the start of another member. */
 #define pb_delta(st, m1, m2) ((int)offsetof(st, m1) - (int)offsetof(st, m2))
-/* Delta from start of structure to member. */
-#define pb_fielddelta_first(st, m1, m2) (offsetof(st, m1))
-/* Delta from end of one field to start of another field. */
-#define pb_fielddelta_other(st, m1, m2) (offsetof(st, m1) - offsetof(st, m2) - pb_membersize(st, m2))
-/* Choose between pb_fielddelta_first and pb_fielddelta_other (backwards compatibility) */
-#define pb_fielddelta_choose(st, m1, m2) (int)(offsetof(st, m1) == offsetof(st, m2) \
-                                  ? pb_fielddelta_first(st, m1, m2) \
-                                  : pb_fielddelta_other(st, m1, m2))
+/* Marks the end of the field list */
 #define PB_LAST_FIELD {0,(pb_type_t) 0,0,0,0,0,0}
 
+/* Macros for filling in the data_offset field */
+/* data_offset for first field in a message */
+#define PB_DATAOFFSET_FIRST(st, m1, m2) (offsetof(st, m1))
+/* data_offset for subsequent fields */
+#define PB_DATAOFFSET_OTHER(st, m1, m2) (offsetof(st, m1) - offsetof(st, m2) - pb_membersize(st, m2))
+/* Choose first/other based on m1 == m2 (deprecated, remains for backwards compatibility) */
+#define PB_DATAOFFSET_CHOOSE(st, m1, m2) (int)(offsetof(st, m1) == offsetof(st, m2) \
+                                  ? PB_DATAOFFSET_FIRST(st, m1, m2) \
+                                  : PB_DATAOFFSET_OTHER(st, m1, m2))
+
 /* Required fields are the simplest. They just have delta (padding) from
  * previous field end, and the size of the field. Pointer is used for
  * submessages and default values.
@@ -359,6 +411,22 @@ struct _pb_extension_t {
     pb_membersize(st, m[0]), \
     pb_arraysize(st, m), ptr}
 
+/* Allocated fields carry the size of the actual data, not the pointer */
+#define PB_REQUIRED_POINTER(tag, st, m, fd, ltype, ptr) \
+    {tag, PB_ATYPE_POINTER | PB_HTYPE_REQUIRED | ltype, \
+    fd, 0, pb_membersize(st, m[0]), 0, ptr}
+
+/* Optional fields don't need a has_ variable, as information would be redundant */
+#define PB_OPTIONAL_POINTER(tag, st, m, fd, ltype, ptr) \
+    {tag, PB_ATYPE_POINTER | PB_HTYPE_OPTIONAL | ltype, \
+    fd, 0, pb_membersize(st, m[0]), 0, ptr}
+
+/* Repeated fields have a _count field and a pointer to array of pointers */
+#define PB_REPEATED_POINTER(tag, st, m, fd, ltype, ptr) \
+    {tag, PB_ATYPE_POINTER | PB_HTYPE_REPEATED | ltype, \
+    fd, pb_delta(st, m ## _count, m), \
+    pb_membersize(st, m[0]), 0, ptr}
+
 /* Callbacks are much like required fields except with special datatype. */
 #define PB_REQUIRED_CALLBACK(tag, st, m, fd, ltype, ptr) \
     {tag, PB_ATYPE_CALLBACK | PB_HTYPE_REQUIRED | ltype, \
@@ -399,8 +467,8 @@ struct _pb_extension_t {
 #define PB_LTYPE_MAP_SINT32     PB_LTYPE_SVARINT
 #define PB_LTYPE_MAP_SINT64     PB_LTYPE_SVARINT
 #define PB_LTYPE_MAP_STRING     PB_LTYPE_STRING
-#define PB_LTYPE_MAP_UINT32     PB_LTYPE_VARINT
-#define PB_LTYPE_MAP_UINT64     PB_LTYPE_VARINT
+#define PB_LTYPE_MAP_UINT32     PB_LTYPE_UVARINT
+#define PB_LTYPE_MAP_UINT64     PB_LTYPE_UVARINT
 #define PB_LTYPE_MAP_EXTENSION  PB_LTYPE_EXTENSION
 
 /* This is the actual macro used in field descriptions.
@@ -411,24 +479,16 @@ struct _pb_extension_t {
  *                 SINT32, SINT64, STRING, UINT32, UINT64 or EXTENSION
  * - Field rules:  REQUIRED, OPTIONAL or REPEATED
  * - Allocation:   STATIC or CALLBACK
+ * - Placement: FIRST or OTHER, depending on if this is the first field in structure.
  * - Message name
  * - Field name
  * - Previous field name (or field name again for first field)
  * - Pointer to default value or submsg fields.
  */
 
-#define PB_FIELD(tag, type, rules, allocation, message, field, prevfield, ptr) \
-    PB_ ## rules ## _ ## allocation(tag, message, field, \
-        pb_fielddelta_choose(message, field, prevfield), \
-        PB_LTYPE_MAP_ ## type, ptr)
-
-/* This is a new version of the macro used by nanopb generator from
- * version 0.2.3 onwards. It avoids the use of a ternary expression in
- * the initialization, which confused some compilers.
- */
-#define PB_FIELD2(tag, type, rules, allocation, message, field, prevfield, pos, ptr) \
-    PB_ ## rules ## _ ## allocation(tag, message, field, \
-        pb_fielddelta_ ## pos(message, field, prevfield), \
+#define PB_FIELD(tag, type, rules, allocation, placement, message, field, prevfield, ptr) \
+        PB_ ## rules ## _ ## allocation(tag, message, field, \
+        PB_DATAOFFSET_ ## placement(message, field, prevfield), \
         PB_LTYPE_MAP_ ## type, ptr)
 
 
@@ -439,7 +499,11 @@ struct _pb_extension_t {
  * messages if not used.
  */
 #ifdef PB_NO_ERRMSG
-#define PB_RETURN_ERROR(stream,msg) return false
+#define PB_RETURN_ERROR(stream,msg) \
+    do {\
+        PB_UNUSED(stream); \
+        return false; \
+    } while(0)
 #define PB_GET_ERROR(stream) "(errmsg disabled)"
 #else
 #define PB_RETURN_ERROR(stream,msg) \

pb_common.c

@@ -0,0 +1,90 @@
+/* pb_common.c: Common support functions for pb_encode.c and pb_decode.c.
+ *
+ * 2014 Petteri Aimonen <jpa@kapsi.fi>
+ */
+
+#include "pb_common.h"
+
+bool pb_field_iter_begin(pb_field_iter_t *iter, const pb_field_t *fields, void *dest_struct)
+{
+    iter->start = fields;
+    iter->pos = fields;
+    iter->required_field_index = 0;
+    iter->dest_struct = dest_struct;
+    iter->pData = (char*)dest_struct + iter->pos->data_offset;
+    iter->pSize = (char*)iter->pData + iter->pos->size_offset;
+    
+    return (iter->pos->tag != 0);
+}
+
+bool pb_field_iter_next(pb_field_iter_t *iter)
+{
+    const pb_field_t *prev_field = iter->pos;
+
+    if (prev_field->tag == 0)
+    {
+        /* Handle empty message types, where the first field is already the terminator.
+         * In other cases, the iter->pos never points to the terminator. */
+        return false;
+    }
+    
+    iter->pos++;
+    
+    if (iter->pos->tag == 0)
+    {
+        /* Wrapped back to beginning, reinitialize */
+        (void)pb_field_iter_begin(iter, iter->start, iter->dest_struct);
+        return false;
+    }
+    else
+    {
+        /* Increment the pointers based on previous field size */
+        size_t prev_size = prev_field->data_size;
+    
+        if (PB_ATYPE(prev_field->type) == PB_ATYPE_STATIC &&
+            PB_HTYPE(prev_field->type) == PB_HTYPE_REPEATED)
+        {
+            /* In static arrays, the data_size tells the size of a single entry and
+             * array_size is the number of entries */
+            prev_size *= prev_field->array_size;
+        }
+        else if (PB_ATYPE(prev_field->type) == PB_ATYPE_POINTER)
+        {
+            /* Pointer fields always have a constant size in the main structure.
+             * The data_size only applies to the dynamically allocated area. */
+            prev_size = sizeof(void*);
+        }
+        
+        if (PB_HTYPE(prev_field->type) == PB_HTYPE_REQUIRED)
+        {
+            /* Count the required fields, in order to check their presence in the
+             * decoder. */
+            iter->required_field_index++;
+        }
+    
+        iter->pData = (char*)iter->pData + prev_size + iter->pos->data_offset;
+        iter->pSize = (char*)iter->pData + iter->pos->size_offset;
+        return true;
+    }
+}
+
+bool pb_field_iter_find(pb_field_iter_t *iter, uint32_t tag)
+{
+    const pb_field_t *start = iter->pos;
+    
+    do {
+        if (iter->pos->tag == tag &&
+            PB_LTYPE(iter->pos->type) != PB_LTYPE_EXTENSION)
+        {
+            /* Found the wanted field */
+            return true;
+        }
+        
+        (void)pb_field_iter_next(iter);
+    } while (iter->pos != start);
+    
+    /* Searched all the way back to start, and found nothing. */
+    return false;
+}
+
+

pb_common.h

@@ -0,0 +1,42 @@
+/* pb_common.h: Common support functions for pb_encode.c and pb_decode.c.
+ * These functions are rarely needed by applications directly.
+ */
+
+#ifndef PB_COMMON_H_INCLUDED
+#define PB_COMMON_H_INCLUDED
+
+#include "pb.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/* Iterator for pb_field_t list */
+struct pb_field_iter_s {
+    const pb_field_t *start;       /* Start of the pb_field_t array */
+    const pb_field_t *pos;         /* Current position of the iterator */
+    unsigned required_field_index; /* Zero-based index that counts only the required fields */
+    void *dest_struct;             /* Pointer to start of the structure */
+    void *pData;                   /* Pointer to current field value */
+    void *pSize;                   /* Pointer to count/has field */
+};
+typedef struct pb_field_iter_s pb_field_iter_t;
+
+/* Initialize the field iterator structure to beginning.
+ * Returns false if the message type is empty. */
+bool pb_field_iter_begin(pb_field_iter_t *iter, const pb_field_t *fields, void *dest_struct);
+
+/* Advance the iterator to the next field.
+ * Returns false when the iterator wraps back to the first field. */
+bool pb_field_iter_next(pb_field_iter_t *iter);
+
+/* Advance the iterator until it points at a field with the given tag.
+ * Returns false if no such field exists. */
+bool pb_field_iter_find(pb_field_iter_t *iter, uint32_t tag);
+
+#ifdef __cplusplus
+} /* extern "C" */
+#endif
+
+#endif
+

pb_decode.c

@@ -3,25 +3,58 @@
  * 2011 Petteri Aimonen <jpa@kapsi.fi>
  */
 
-/* The warn_unused_result attribute appeared first in gcc-3.4.0 */
+/* Use the GCC warn_unused_result attribute to check that all return values
+ * are propagated correctly. On other compilers and gcc before 3.4.0 just
+ * ignore the annotation.
+ */
 #if !defined(__GNUC__) || ( __GNUC__ < 3) || (__GNUC__ == 3 && __GNUC_MINOR__ < 4)
     #define checkreturn
 #else
-    /* Verify that we remember to check all return values for proper error propagation */
     #define checkreturn __attribute__((warn_unused_result))
 #endif
 
-#define NANOPB_INTERNALS
 #include "pb.h"
 #include "pb_decode.h"
+#include "pb_common.h"
+
+/**************************************
+ * Declarations internal to this file *
+ **************************************/
 
 typedef bool (*pb_decoder_t)(pb_istream_t *stream, const pb_field_t *field, void *dest) checkreturn;
 
+static bool checkreturn buf_read(pb_istream_t *stream, uint8_t *buf, size_t count);
+static bool checkreturn pb_decode_varint32(pb_istream_t *stream, uint32_t *dest);
+static bool checkreturn read_raw_value(pb_istream_t *stream, pb_wire_type_t wire_type, uint8_t *buf, size_t *size);
+static bool checkreturn decode_static_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iter_t *iter);
+static bool checkreturn decode_callback_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iter_t *iter);
+static bool checkreturn decode_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iter_t *iter);
+static bool checkreturn default_extension_decoder(pb_istream_t *stream, pb_extension_t *extension, uint32_t tag, pb_wire_type_t wire_type);
+static bool checkreturn decode_extension(pb_istream_t *stream, uint32_t tag, pb_wire_type_t wire_type, pb_field_iter_t *iter);
+static bool checkreturn find_extension_field(pb_field_iter_t *iter);
+static void pb_message_set_to_defaults(const pb_field_t fields[], void *dest_struct);
+static bool checkreturn pb_dec_varint(pb_istream_t *stream, const pb_field_t *field, void *dest);
+static bool checkreturn pb_dec_uvarint(pb_istream_t *stream, const pb_field_t *field, void *dest);
+static bool checkreturn pb_dec_svarint(pb_istream_t *stream, const pb_field_t *field, void *dest);
+static bool checkreturn pb_dec_fixed32(pb_istream_t *stream, const pb_field_t *field, void *dest);
+static bool checkreturn pb_dec_fixed64(pb_istream_t *stream, const pb_field_t *field, void *dest);
+static bool checkreturn pb_dec_bytes(pb_istream_t *stream, const pb_field_t *field, void *dest);
+static bool checkreturn pb_dec_string(pb_istream_t *stream, const pb_field_t *field, void *dest);
+static bool checkreturn pb_dec_submessage(pb_istream_t *stream, const pb_field_t *field, void *dest);
+static bool checkreturn pb_skip_varint(pb_istream_t *stream);
+static bool checkreturn pb_skip_string(pb_istream_t *stream);
+
+#ifdef PB_ENABLE_MALLOC
+static bool checkreturn allocate_field(pb_istream_t *stream, void *pData, size_t data_size, size_t array_size);
+static void pb_release_single_field(const pb_field_iter_t *iter);
+#endif
+
 /* --- Function pointers to field decoders ---
  * Order in the array must match pb_action_t LTYPE numbering.
  */
 static const pb_decoder_t PB_DECODERS[PB_LTYPES_COUNT] = {
     &pb_dec_varint,
+    &pb_dec_uvarint,
     &pb_dec_svarint,
     &pb_dec_fixed32,
     &pb_dec_fixed64,
@@ -32,9 +65,9 @@ static const pb_decoder_t PB_DECODERS[PB_LTYPES_COUNT] = {
     NULL /* extensions */
 };
 
-/**************
- * pb_istream *
- **************/
+/*******************************
+ * pb_istream_t implementation *
+ *******************************/
 
 static bool checkreturn buf_read(pb_istream_t *stream, uint8_t *buf, size_t count)
 {
@@ -84,6 +117,26 @@ bool checkreturn pb_read(pb_istream_t *stream, uint8_t *buf, size_t count)
     return true;
 }
 
+/* Read a single byte from input stream. buf may not be NULL.
+ * This is an optimization for the varint decoding. */
+static bool checkreturn pb_readbyte(pb_istream_t *stream, uint8_t *buf)
+{
+    if (stream->bytes_left == 0)
+        PB_RETURN_ERROR(stream, "end-of-stream");
+
+#ifndef PB_BUFFER_ONLY
+    if (!stream->callback(stream, buf, 1))
+        PB_RETURN_ERROR(stream, "io error");
+#else
+    *buf = *(uint8_t*)stream->state;
+    stream->state = (uint8_t*)stream->state + 1;
+#endif
+
+    stream->bytes_left--;
+    
+    return true;    
+}
+
 pb_istream_t pb_istream_from_buffer(uint8_t *buf, size_t bufsize)
 {
     pb_istream_t stream;
@@ -109,10 +162,10 @@ static bool checkreturn pb_decode_varint32(pb_istream_t *stream, uint32_t *dest)
     uint8_t byte;
     uint32_t result;
     
-    if (!pb_read(stream, &byte, 1))
+    if (!pb_readbyte(stream, &byte))
         return false;
     
-    if (!(byte & 0x80))
+    if ((byte & 0x80) == 0)
     {
         /* Quick case, 1 byte value */
         result = byte;
@@ -128,7 +181,7 @@ static bool checkreturn pb_decode_varint32(pb_istream_t *stream, uint32_t *dest)
             if (bitpos >= 32)
                 PB_RETURN_ERROR(stream, "varint overflow");
             
-            if (!pb_read(stream, &byte, 1))
+            if (!pb_readbyte(stream, &byte))
                 return false;
             
             result |= (uint32_t)(byte & 0x7F) << bitpos;
@@ -151,7 +204,7 @@ bool checkreturn pb_decode_varint(pb_istream_t *stream, uint64_t *dest)
         if (bitpos >= 64)
             PB_RETURN_ERROR(stream, "varint overflow");
         
-        if (!pb_read(stream, &byte, 1))
+        if (!pb_readbyte(stream, &byte))
             return false;
 
         result |= (uint64_t)(byte & 0x7F) << bitpos;
@@ -277,82 +330,11 @@ void pb_close_string_substream(pb_istream_t *stream, pb_istream_t *substream)
 #endif
 }
 
-/* Iterator for pb_field_t list */
-typedef struct {
-    const pb_field_t *start; /* Start of the pb_field_t array */
-    const pb_field_t *pos; /* Current position of the iterator */
-    unsigned field_index; /* Zero-based index of the field. */
-    unsigned required_field_index; /* Zero-based index that counts only the required fields */
-    void *dest_struct; /* Pointer to the destination structure to decode to */
-    void *pData; /* Pointer where to store current field value */
-    void *pSize; /* Pointer where to store the size of current array field */
-} pb_field_iterator_t;
-
-static void pb_field_init(pb_field_iterator_t *iter, const pb_field_t *fields, void *dest_struct)
-{
-    iter->start = iter->pos = fields;
-    iter->field_index = 0;
-    iter->required_field_index = 0;
-    iter->pData = (char*)dest_struct + iter->pos->data_offset;
-    iter->pSize = (char*)iter->pData + iter->pos->size_offset;
-    iter->dest_struct = dest_struct;
-}
-
-static bool pb_field_next(pb_field_iterator_t *iter)
-{
-    bool notwrapped = true;
-    size_t prev_size = iter->pos->data_size;
-    
-    if (PB_ATYPE(iter->pos->type) == PB_ATYPE_STATIC &&
-        PB_HTYPE(iter->pos->type) == PB_HTYPE_REPEATED)
-    {
-        prev_size *= iter->pos->array_size;
-    }
-    
-    if (iter->pos->tag == 0)
-        return false; /* Only happens with empty message types */
-    
-    if (PB_HTYPE(iter->pos->type) == PB_HTYPE_REQUIRED)
-        iter->required_field_index++;
-    
-    iter->pos++;
-    iter->field_index++;
-    if (iter->pos->tag == 0)
-    {
-        iter->pos = iter->start;
-        iter->field_index = 0;
-        iter->required_field_index = 0;
-        iter->pData = iter->dest_struct;
-        prev_size = 0;
-        notwrapped = false;
-    }
-    
-    iter->pData = (char*)iter->pData + prev_size + iter->pos->data_offset;
-    iter->pSize = (char*)iter->pData + iter->pos->size_offset;
-    return notwrapped;
-}
-
-static bool checkreturn pb_field_find(pb_field_iterator_t *iter, uint32_t tag)
-{
-    unsigned start = iter->field_index;
-    
-    do {
-        if (iter->pos->tag == tag &&
-            PB_LTYPE(iter->pos->type) != PB_LTYPE_EXTENSION)
-        {
-            return true;
-        }
-        pb_field_next(iter);
-    } while (iter->field_index != start);
-    
-    return false;
-}
-
 /*************************
  * Decode a single field *
  *************************/
 
-static bool checkreturn decode_static_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iterator_t *iter)
+static bool checkreturn decode_static_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iter_t *iter)
 {
     pb_type_t type;
     pb_decoder_t func;
@@ -375,12 +357,12 @@ static bool checkreturn decode_static_field(pb_istream_t *stream, pb_wire_type_t
             {
                 /* Packed array */
                 bool status = true;
-                size_t *size = (size_t*)iter->pSize;
+                pb_size_t *size = (pb_size_t*)iter->pSize;
                 pb_istream_t substream;
                 if (!pb_make_string_substream(stream, &substream))
                     return false;
                 
-                while (substream.bytes_left && *size < iter->pos->array_size)
+                while (substream.bytes_left > 0 && *size < iter->pos->array_size)
                 {
                     void *pItem = (uint8_t*)iter->pData + iter->pos->data_size * (*size);
                     if (!func(&substream, iter->pos, pItem))
@@ -400,7 +382,7 @@ static bool checkreturn decode_static_field(pb_istream_t *stream, pb_wire_type_t
             else
             {
                 /* Repeated field */
-                size_t *size = (size_t*)iter->pSize;
+                pb_size_t *size = (pb_size_t*)iter->pSize;
                 void *pItem = (uint8_t*)iter->pData + iter->pos->data_size * (*size);
                 if (*size >= iter->pos->array_size)
                     PB_RETURN_ERROR(stream, "array overflow");
@@ -414,7 +396,178 @@ static bool checkreturn decode_static_field(pb_istream_t *stream, pb_wire_type_t
     }
 }
 
-static bool checkreturn decode_callback_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iterator_t *iter)
+#ifdef PB_ENABLE_MALLOC
+/* Allocate storage for the field and store the pointer at iter->pData.
+ * array_size is the number of entries to reserve in an array.
+ * Zero size is not allowed, use pb_free() for releasing.
+ */
+static bool checkreturn allocate_field(pb_istream_t *stream, void *pData, size_t data_size, size_t array_size)
+{    
+    void *ptr = *(void**)pData;
+    
+    if (data_size == 0 || array_size == 0)
+        PB_RETURN_ERROR(stream, "invalid size");
+    
+    /* Check for multiplication overflows.
+     * This code avoids the costly division if the sizes are small enough.
+     * Multiplication is safe as long as only half of bits are set
+     * in either multiplicand.
+     */
+    {
+        const size_t check_limit = (size_t)1 << (sizeof(size_t) * 4);
+        if (data_size >= check_limit || array_size >= check_limit)
+        {
+            const size_t size_max = (size_t)-1;
+            if (size_max / array_size < data_size)
+            {
+                PB_RETURN_ERROR(stream, "size too large");
+            }
+        }
+    }
+    
+    /* Allocate new or expand previous allocation */
+    /* Note: on failure the old pointer will remain in the structure,
+     * the message must be freed by caller also on error return. */
+    ptr = pb_realloc(ptr, array_size * data_size);
+    if (ptr == NULL)
+        PB_RETURN_ERROR(stream, "realloc failed");
+    
+    *(void**)pData = ptr;
+    return true;
+}
+
+/* Clear a newly allocated item in case it contains a pointer, or is a submessage. */
+static void initialize_pointer_field(void *pItem, pb_field_iter_t *iter)
+{
+    if (PB_LTYPE(iter->pos->type) == PB_LTYPE_STRING ||
+        PB_LTYPE(iter->pos->type) == PB_LTYPE_BYTES)
+    {
+        *(void**)pItem = NULL;
+    }
+    else if (PB_LTYPE(iter->pos->type) == PB_LTYPE_SUBMESSAGE)
+    {
+        pb_message_set_to_defaults((const pb_field_t *) iter->pos->ptr, pItem);
+    }
+}
+#endif
+
+static bool checkreturn decode_pointer_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iter_t *iter)
+{
+#ifndef PB_ENABLE_MALLOC
+    PB_UNUSED(wire_type);
+    PB_UNUSED(iter);
+    PB_RETURN_ERROR(stream, "no malloc support");
+#else
+    pb_type_t type;
+    pb_decoder_t func;
+    
+    type = iter->pos->type;
+    func = PB_DECODERS[PB_LTYPE(type)];
+    
+    switch (PB_HTYPE(type))
+    {
+        case PB_HTYPE_REQUIRED:
+        case PB_HTYPE_OPTIONAL:
+            if (PB_LTYPE(type) == PB_LTYPE_SUBMESSAGE &&
+                *(void**)iter->pData != NULL)
+            {
+                /* Duplicate field, have to release the old allocation first. */
+                pb_release_single_field(iter);
+            }
+        
+            if (PB_LTYPE(type) == PB_LTYPE_STRING ||
+                PB_LTYPE(type) == PB_LTYPE_BYTES)
+            {
+                return func(stream, iter->pos, iter->pData);
+            }
+            else
+            {
+                if (!allocate_field(stream, iter->pData, iter->pos->data_size, 1))
+                    return false;
+                
+                initialize_pointer_field(*(void**)iter->pData, iter);
+                return func(stream, iter->pos, *(void**)iter->pData);
+            }
+    
+        case PB_HTYPE_REPEATED:
+            if (wire_type == PB_WT_STRING
+                && PB_LTYPE(type) <= PB_LTYPE_LAST_PACKABLE)
+            {
+                /* Packed array, multiple items come in at once. */
+                bool status = true;
+                pb_size_t *size = (pb_size_t*)iter->pSize;
+                size_t allocated_size = *size;
+                void *pItem;
+                pb_istream_t substream;
+                
+                if (!pb_make_string_substream(stream, &substream))
+                    return false;
+                
+                while (substream.bytes_left)
+                {
+                    if ((size_t)*size + 1 > allocated_size)
+                    {
+                        /* Allocate more storage. This tries to guess the
+                         * number of remaining entries. Round the division
+                         * upwards. */
+                        allocated_size += (substream.bytes_left - 1) / iter->pos->data_size + 1;
+                        
+                        if (!allocate_field(&substream, iter->pData, iter->pos->data_size, allocated_size))
+                        {
+                            status = false;
+                            break;
+                        }
+                    }
+
+                    /* Decode the array entry */
+                    pItem = *(uint8_t**)iter->pData + iter->pos->data_size * (*size);
+                    initialize_pointer_field(pItem, iter);
+                    if (!func(&substream, iter->pos, pItem))
+                    {
+                        status = false;
+                        break;
+                    }
+                    
+                    if (*size == PB_SIZE_MAX)
+                    {
+#ifndef PB_NO_ERRMSG
+                        stream->errmsg = "too many array entries";
+#endif
+                        status = false;
+                        break;
+                    }
+                    
+                    (*size)++;
+                }
+                pb_close_string_substream(stream, &substream);
+                
+                return status;
+            }
+            else
+            {
+                /* Normal repeated field, i.e. only one item at a time. */
+                pb_size_t *size = (pb_size_t*)iter->pSize;
+                void *pItem;
+                
+                if (*size == PB_SIZE_MAX)
+                    PB_RETURN_ERROR(stream, "too many array entries");
+                
+                (*size)++;
+                if (!allocate_field(stream, iter->pData, iter->pos->data_size, *size))
+                    return false;
+            
+                pItem = *(uint8_t**)iter->pData + iter->pos->data_size * (*size - 1);
+                initialize_pointer_field(pItem, iter);
+                return func(stream, iter->pos, pItem);
+            }
+            
+        default:
+            PB_RETURN_ERROR(stream, "invalid field type");
+    }
+#endif
+}
+
+static bool checkreturn decode_callback_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iter_t *iter)
 {
     pb_callback_t *pCallback = (pb_callback_t*)iter->pData;
     
@@ -461,13 +614,16 @@ static bool checkreturn decode_callback_field(pb_istream_t *stream, pb_wire_type
     }
 }
 
-static bool checkreturn decode_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iterator_t *iter)
+static bool checkreturn decode_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iter_t *iter)
 {
     switch (PB_ATYPE(iter->pos->type))
     {
         case PB_ATYPE_STATIC:
             return decode_static_field(stream, wire_type, iter);
         
+        case PB_ATYPE_POINTER:
+            return decode_pointer_field(stream, wire_type, iter);
+        
         case PB_ATYPE_CALLBACK:
             return decode_callback_field(stream, wire_type, iter);
         
@@ -478,23 +634,21 @@ static bool checkreturn decode_field(pb_istream_t *stream, pb_wire_type_t wire_t
 
 /* Default handler for extension fields. Expects a pb_field_t structure
  * in extension->type->arg. */
-static bool checkreturn default_extension_handler(pb_istream_t *stream,
+static bool checkreturn default_extension_decoder(pb_istream_t *stream,
     pb_extension_t *extension, uint32_t tag, pb_wire_type_t wire_type)
 {
     const pb_field_t *field = (const pb_field_t*)extension->type->arg;
-    pb_field_iterator_t iter;
-    bool dummy;
+    pb_field_iter_t iter;
     
     if (field->tag != tag)
         return true;
     
-    iter.start = field;
-    iter.pos = field;
-    iter.field_index = 0;
-    iter.required_field_index = 0;
-    iter.dest_struct = extension->dest;
+    /* Fake a field iterator for the extension field.
+     * It is not actually safe to advance this iterator, but decode_field
+     * will not even try to. */
+    (void)pb_field_iter_begin(&iter, field, extension->dest);
     iter.pData = extension->dest;
-    iter.pSize = &dummy;
+    iter.pSize = &extension->found;
     
     return decode_field(stream, wire_type, &iter);
 }
@@ -502,18 +656,18 @@ static bool checkreturn default_extension_handler(pb_istream_t *stream,
 /* Try to decode an unknown field as an extension field. Tries each extension
  * decoder in turn, until one of them handles the field or loop ends. */
 static bool checkreturn decode_extension(pb_istream_t *stream,
-    uint32_t tag, pb_wire_type_t wire_type, pb_field_iterator_t *iter)
+    uint32_t tag, pb_wire_type_t wire_type, pb_field_iter_t *iter)
 {
     pb_extension_t *extension = *(pb_extension_t* const *)iter->pData;
     size_t pos = stream->bytes_left;
     
-    while (extension && pos == stream->bytes_left)
+    while (extension != NULL && pos == stream->bytes_left)
     {
         bool status;
         if (extension->type->decode)
             status = extension->type->decode(stream, extension, tag, wire_type);
         else
-            status = default_extension_handler(stream, extension, tag, wire_type);
+            status = default_extension_decoder(stream, extension, tag, wire_type);
 
         if (!status)
             return false;
@@ -527,15 +681,15 @@ static bool checkreturn decode_extension(pb_istream_t *stream,
 /* Step through the iterator until an extension field is found or until all
  * entries have been checked. There can be only one extension field per
  * message. Returns false if no extension field is found. */
-static bool checkreturn find_extension_field(pb_field_iterator_t *iter)
+static bool checkreturn find_extension_field(pb_field_iter_t *iter)
 {
-    unsigned start = iter->field_index;
+    const pb_field_t *start = iter->pos;
     
     do {
         if (PB_LTYPE(iter->pos->type) == PB_LTYPE_EXTENSION)
             return true;
-        pb_field_next(iter);
-    } while (iter->field_index != start);
+        (void)pb_field_iter_next(iter);
+    } while (iter->pos != start);
     
     return false;
 }
@@ -543,50 +697,63 @@ static bool checkreturn find_extension_field(pb_field_iterator_t *iter)
 /* Initialize message fields to default values, recursively */
 static void pb_message_set_to_defaults(const pb_field_t fields[], void *dest_struct)
 {
-    pb_field_iterator_t iter;
-    pb_field_init(&iter, fields, dest_struct);
+    pb_field_iter_t iter;
+
+    if (!pb_field_iter_begin(&iter, fields, dest_struct))
+        return; /* Empty message type */
     
-    /* Initialize size/has fields and apply default values */
     do
     {
         pb_type_t type;
         type = iter.pos->type;
-    
-        if (iter.pos->tag == 0)
-            continue;
         
         if (PB_ATYPE(type) == PB_ATYPE_STATIC)
         {
-            /* Initialize the size field for optional/repeated fields to 0. */
             if (PB_HTYPE(type) == PB_HTYPE_OPTIONAL)
             {
+                /* Set has_field to false. Still initialize the optional field
+                 * itself also. */
                 *(bool*)iter.pSize = false;
             }
             else if (PB_HTYPE(type) == PB_HTYPE_REPEATED)
             {
-                *(size_t*)iter.pSize = 0;
-                continue; /* Array is empty, no need to initialize contents */
+                /* Set array count to 0, no need to initialize contents. */
+                *(pb_size_t*)iter.pSize = 0;
+                continue;
             }
             
-            /* Initialize field contents to default value */
             if (PB_LTYPE(iter.pos->type) == PB_LTYPE_SUBMESSAGE)
             {
+                /* Initialize submessage to defaults */
                 pb_message_set_to_defaults((const pb_field_t *) iter.pos->ptr, iter.pData);
             }
             else if (iter.pos->ptr != NULL)
             {
+                /* Initialize to default value */
                 memcpy(iter.pData, iter.pos->ptr, iter.pos->data_size);
             }
             else
             {
+                /* Initialize to zeros */
                 memset(iter.pData, 0, iter.pos->data_size);
             }
         }
+        else if (PB_ATYPE(type) == PB_ATYPE_POINTER)
+        {
+            /* Initialize the pointer to NULL. */
+            *(void**)iter.pData = NULL;
+            
+            /* Initialize array count to 0. */
+            if (PB_HTYPE(type) == PB_HTYPE_REPEATED)
+            {
+                *(pb_size_t*)iter.pSize = 0;
+            }
+        }
         else if (PB_ATYPE(type) == PB_ATYPE_CALLBACK)
         {
-            continue; /* Don't overwrite callback */
+            /* Don't overwrite callback */
         }
-    } while (pb_field_next(&iter));
+    } while (pb_field_iter_next(&iter));
 }
 
 /*********************
@@ -595,11 +762,13 @@ static void pb_message_set_to_defaults(const pb_field_t fields[], void *dest_str
 
 bool checkreturn pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct)
 {
-    uint8_t fields_seen[(PB_MAX_REQUIRED_FIELDS + 7) / 8] = {0}; /* Used to check for required fields */
+    uint8_t fields_seen[(PB_MAX_REQUIRED_FIELDS + 7) / 8] = {0, 0, 0, 0, 0, 0, 0, 0};
     uint32_t extension_range_start = 0;
-    pb_field_iterator_t iter;
+    pb_field_iter_t iter;
     
-    pb_field_init(&iter, fields, dest_struct);
+    /* Return value ignored, as empty message types will be correctly handled by
+     * pb_field_iter_find() anyway. */
+    (void)pb_field_iter_begin(&iter, fields, dest_struct);
     
     while (stream->bytes_left)
     {
@@ -615,7 +784,7 @@ bool checkreturn pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[
                 return false;
         }
         
-        if (!pb_field_find(&iter, tag))
+        if (!pb_field_iter_find(&iter, tag))
         {
             /* No match found, check if it matches an extension. */
             if (tag >= extension_range_start)
@@ -668,10 +837,10 @@ bool checkreturn pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[
         do {
             req_field_count = iter.required_field_index;
             last_type = iter.pos->type;
-        } while (pb_field_next(&iter));
+        } while (pb_field_iter_next(&iter));
         
         /* Fixup if last field was also required. */
-        if (PB_HTYPE(last_type) == PB_HTYPE_REQUIRED && iter.pos->tag)
+        if (PB_HTYPE(last_type) == PB_HTYPE_REQUIRED && iter.pos->tag != 0)
             req_field_count++;
         
         /* Check the whole bytes */
@@ -691,8 +860,16 @@ bool checkreturn pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[
 
 bool checkreturn pb_decode(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct)
 {
+    bool status;
     pb_message_set_to_defaults(fields, dest_struct);
-    return pb_decode_noinit(stream, fields, dest_struct);
+    status = pb_decode_noinit(stream, fields, dest_struct);
+    
+#ifdef PB_ENABLE_MALLOC
+    if (!status)
+        pb_release(fields, dest_struct);
+#endif
+    
+    return status;
 }
 
 bool pb_decode_delimited(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct)
@@ -708,6 +885,70 @@ bool pb_decode_delimited(pb_istream_t *stream, const pb_field_t fields[], void *
     return status;
 }
 
+#ifdef PB_ENABLE_MALLOC
+static void pb_release_single_field(const pb_field_iter_t *iter)
+{
+    pb_type_t type;
+    type = iter->pos->type;
+
+    if (PB_ATYPE(type) == PB_ATYPE_POINTER)
+    {
+        if (PB_HTYPE(type) == PB_HTYPE_REPEATED &&
+            (PB_LTYPE(type) == PB_LTYPE_STRING ||
+             PB_LTYPE(type) == PB_LTYPE_BYTES))
+        {
+            /* Release entries in repeated string or bytes array */
+            void **pItem = *(void***)iter->pData;
+            pb_size_t count = *(pb_size_t*)iter->pSize;
+            while (count--)
+            {
+                pb_free(*pItem);
+                *pItem++ = NULL;
+            }
+            *(pb_size_t*)iter->pSize = 0;
+        }
+        else if (PB_LTYPE(type) == PB_LTYPE_SUBMESSAGE)
+        {
+            /* Release fields in submessages */
+            void *pItem = *(void**)iter->pData;
+            if (pItem)
+            {
+                pb_size_t count = 1;
+                
+                if (PB_HTYPE(type) == PB_HTYPE_REPEATED)
+                {
+                    count = *(pb_size_t*)iter->pSize;
+                    *(pb_size_t*)iter->pSize = 0;
+                }
+                
+                while (count--)
+                {
+                    pb_release((const pb_field_t*)iter->pos->ptr, pItem);
+                    pItem = (uint8_t*)pItem + iter->pos->data_size;
+                }
+            }
+        }
+        
+        /* Release main item */
+        pb_free(*(void**)iter->pData);
+        *(void**)iter->pData = NULL;
+    }
+}
+
+void pb_release(const pb_field_t fields[], void *dest_struct)
+{
+    pb_field_iter_t iter;
+    
+    if (!pb_field_iter_begin(&iter, fields, dest_struct))
+        return; /* Empty message type */
+    
+    do
+    {
+        pb_release_single_field(&iter);
+    } while (pb_field_iter_next(&iter));
+}
+#endif
+
 /* Field decoders */
 
 bool pb_decode_svarint(pb_istream_t *stream, int64_t *dest)
@@ -766,7 +1007,25 @@ bool pb_decode_fixed64(pb_istream_t *stream, void *dest)
     #endif   
 }
 
-bool checkreturn pb_dec_varint(pb_istream_t *stream, const pb_field_t *field, void *dest)
+static bool checkreturn pb_dec_varint(pb_istream_t *stream, const pb_field_t *field, void *dest)
+{
+    uint64_t value;
+    if (!pb_decode_varint(stream, &value))
+        return false;
+    
+    switch (field->data_size)
+    {
+        case 1: *(int8_t*)dest = (int8_t)value; break;
+        case 2: *(int16_t*)dest = (int16_t)value; break;
+        case 4: *(int32_t*)dest = (int32_t)value; break;
+        case 8: *(int64_t*)dest = (int64_t)value; break;
+        default: PB_RETURN_ERROR(stream, "invalid data_size");
+    }
+    
+    return true;
+}
+
+static bool checkreturn pb_dec_uvarint(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
     uint64_t value;
     if (!pb_decode_varint(stream, &value))
@@ -774,8 +1033,6 @@ bool checkreturn pb_dec_varint(pb_istream_t *stream, const pb_field_t *field, vo
     
     switch (field->data_size)
     {
-        case 1: *(uint8_t*)dest = (uint8_t)value; break;
-        case 2: *(uint16_t*)dest = (uint16_t)value; break;
         case 4: *(uint32_t*)dest = (uint32_t)value; break;
         case 8: *(uint64_t*)dest = value; break;
         default: PB_RETURN_ERROR(stream, "invalid data_size");
@@ -784,7 +1041,7 @@ bool checkreturn pb_dec_varint(pb_istream_t *stream, const pb_field_t *field, vo
     return true;
 }
 
-bool checkreturn pb_dec_svarint(pb_istream_t *stream, const pb_field_t *field, void *dest)
+static bool checkreturn pb_dec_svarint(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
     int64_t value;
     if (!pb_decode_svarint(stream, &value))
@@ -800,51 +1057,91 @@ bool checkreturn pb_dec_svarint(pb_istream_t *stream, const pb_field_t *field, v
     return true;
 }
 
-bool checkreturn pb_dec_fixed32(pb_istream_t *stream, const pb_field_t *field, void *dest)
+static bool checkreturn pb_dec_fixed32(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
-    UNUSED(field);
+    PB_UNUSED(field);
     return pb_decode_fixed32(stream, dest);
 }
 
-bool checkreturn pb_dec_fixed64(pb_istream_t *stream, const pb_field_t *field, void *dest)
+static bool checkreturn pb_dec_fixed64(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
-    UNUSED(field);
+    PB_UNUSED(field);
     return pb_decode_fixed64(stream, dest);
 }
 
-bool checkreturn pb_dec_bytes(pb_istream_t *stream, const pb_field_t *field, void *dest)
-{
-    pb_bytes_array_t *x = (pb_bytes_array_t*)dest;
-    
-    uint32_t temp;
-    if (!pb_decode_varint32(stream, &temp))
-        return false;
-    x->size = temp;
-    
-    /* Check length, noting the space taken by the size_t header. */
-    if (x->size > field->data_size - offsetof(pb_bytes_array_t, bytes))
-        PB_RETURN_ERROR(stream, "bytes overflow");
-    
-    return pb_read(stream, x->bytes, x->size);
-}
-
-bool checkreturn pb_dec_string(pb_istream_t *stream, const pb_field_t *field, void *dest)
+static bool checkreturn pb_dec_bytes(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
     uint32_t size;
+    size_t alloc_size;
+    pb_bytes_array_t *bdest;
+    
+    if (!pb_decode_varint32(stream, &size))
+        return false;
+    
+    if (size > PB_SIZE_MAX)
+        PB_RETURN_ERROR(stream, "bytes overflow");
+    
+    alloc_size = PB_BYTES_ARRAY_T_ALLOCSIZE(size);
+    if (size > alloc_size)
+        PB_RETURN_ERROR(stream, "size too large");
+    
+    if (PB_ATYPE(field->type) == PB_ATYPE_POINTER)
+    {
+#ifndef PB_ENABLE_MALLOC
+        PB_RETURN_ERROR(stream, "no malloc support");
+#else
+        if (!allocate_field(stream, dest, alloc_size, 1))
+            return false;
+        bdest = *(pb_bytes_array_t**)dest;
+#endif
+    }
+    else
+    {
+        if (alloc_size > field->data_size)
+            PB_RETURN_ERROR(stream, "bytes overflow");
+        bdest = (pb_bytes_array_t*)dest;
+    }
+
+    bdest->size = (pb_size_t)size;
+    return pb_read(stream, bdest->bytes, size);
+}
+
+static bool checkreturn pb_dec_string(pb_istream_t *stream, const pb_field_t *field, void *dest)
+{
+    uint32_t size;
+    size_t alloc_size;
     bool status;
     if (!pb_decode_varint32(stream, &size))
         return false;
     
-    /* Check length, noting the null terminator */
-    if (size + 1 > field->data_size)
-        PB_RETURN_ERROR(stream, "string overflow");
+    /* Space for null terminator */
+    alloc_size = size + 1;
+    
+    if (alloc_size < size)
+        PB_RETURN_ERROR(stream, "size too large");
+    
+    if (PB_ATYPE(field->type) == PB_ATYPE_POINTER)
+    {
+#ifndef PB_ENABLE_MALLOC
+        PB_RETURN_ERROR(stream, "no malloc support");
+#else
+        if (!allocate_field(stream, dest, alloc_size, 1))
+            return false;
+        dest = *(void**)dest;
+#endif
+    }
+    else
+    {
+        if (alloc_size > field->data_size)
+            PB_RETURN_ERROR(stream, "string overflow");
+    }
     
     status = pb_read(stream, (uint8_t*)dest, size);
     *((uint8_t*)dest + size) = 0;
     return status;
 }
 
-bool checkreturn pb_dec_submessage(pb_istream_t *stream, const pb_field_t *field, void *dest)
+static bool checkreturn pb_dec_submessage(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
     bool status;
     pb_istream_t substream;

pb_decode.h

@@ -3,8 +3,8 @@
  * field descriptions created by nanopb_generator.py.
  */
 
-#ifndef _PB_DECODE_H_
-#define _PB_DECODE_H_
+#ifndef PB_DECODE_H_INCLUDED
+#define PB_DECODE_H_INCLUDED
 
 #include "pb.h"
 
@@ -25,7 +25,7 @@ extern "C" {
  *    is different than from the main stream. Don't use bytes_left to compute
  *    any pointers.
  */
-struct _pb_istream_t
+struct pb_istream_s
 {
 #ifdef PB_BUFFER_ONLY
     /* Callback pointer is not used in buffer-only configuration.
@@ -73,6 +73,9 @@ bool pb_decode(pb_istream_t *stream, const pb_field_t fields[], void *dest_struc
  *
  * This can also be used for 'merging' two messages, i.e. update only the
  * fields that exist in the new message.
+ *
+ * Note: If this function returns with an error, it will not release any
+ * dynamically allocated fields. You will need to call pb_release() yourself.
  */
 bool pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct);
 
@@ -82,6 +85,14 @@ bool pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[], void *des
  */
 bool pb_decode_delimited(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct);
 
+#ifdef PB_ENABLE_MALLOC
+/* Release any allocated pointer fields. If you use dynamic allocation, you should
+ * call this for any successfully decoded message when you are done with it. If
+ * pb_decode() returns with an error, the message is already released.
+ */
+void pb_release(const pb_field_t fields[], void *dest_struct);
+#endif
+
 
 /**************************************
  * Functions for manipulating streams *
@@ -131,25 +142,6 @@ bool pb_decode_fixed64(pb_istream_t *stream, void *dest);
 bool pb_make_string_substream(pb_istream_t *stream, pb_istream_t *substream);
 void pb_close_string_substream(pb_istream_t *stream, pb_istream_t *substream);
 
-
-/*******************************
- * Internal / legacy functions *
- *******************************/
-
-#ifdef NANOPB_INTERNALS
-bool pb_dec_varint(pb_istream_t *stream, const pb_field_t *field, void *dest);
-bool pb_dec_svarint(pb_istream_t *stream, const pb_field_t *field, void *dest);
-bool pb_dec_fixed32(pb_istream_t *stream, const pb_field_t *field, void *dest);
-bool pb_dec_fixed64(pb_istream_t *stream, const pb_field_t *field, void *dest);
-
-bool pb_dec_bytes(pb_istream_t *stream, const pb_field_t *field, void *dest);
-bool pb_dec_string(pb_istream_t *stream, const pb_field_t *field, void *dest);
-bool pb_dec_submessage(pb_istream_t *stream, const pb_field_t *field, void *dest);
-
-bool pb_skip_varint(pb_istream_t *stream);
-bool pb_skip_string(pb_istream_t *stream);
-#endif
-
 #ifdef __cplusplus
 } /* extern "C" */
 #endif

pb_encode.c

@@ -3,25 +3,45 @@
  * 2011 Petteri Aimonen <jpa@kapsi.fi>
  */
 
-#define NANOPB_INTERNALS
 #include "pb.h"
 #include "pb_encode.h"
+#include "pb_common.h"
 
-/* The warn_unused_result attribute appeared first in gcc-3.4.0 */
+/* Use the GCC warn_unused_result attribute to check that all return values
+ * are propagated correctly. On other compilers and gcc before 3.4.0 just
+ * ignore the annotation.
+ */
 #if !defined(__GNUC__) || ( __GNUC__ < 3) || (__GNUC__ == 3 && __GNUC_MINOR__ < 4)
     #define checkreturn
 #else
-    /* Verify that we remember to check all return values for proper error propagation */
     #define checkreturn __attribute__((warn_unused_result))
 #endif
 
+/**************************************
+ * Declarations internal to this file *
+ **************************************/
 typedef bool (*pb_encoder_t)(pb_ostream_t *stream, const pb_field_t *field, const void *src) checkreturn;
 
+static bool checkreturn buf_write(pb_ostream_t *stream, const uint8_t *buf, size_t count);
+static bool checkreturn encode_array(pb_ostream_t *stream, const pb_field_t *field, const void *pData, size_t count, pb_encoder_t func);
+static bool checkreturn encode_field(pb_ostream_t *stream, const pb_field_t *field, const void *pData);
+static bool checkreturn default_extension_encoder(pb_ostream_t *stream, const pb_extension_t *extension);
+static bool checkreturn encode_extension_field(pb_ostream_t *stream, const pb_field_t *field, const void *pData);
+static bool checkreturn pb_enc_varint(pb_ostream_t *stream, const pb_field_t *field, const void *src);
+static bool checkreturn pb_enc_uvarint(pb_ostream_t *stream, const pb_field_t *field, const void *src);
+static bool checkreturn pb_enc_svarint(pb_ostream_t *stream, const pb_field_t *field, const void *src);
+static bool checkreturn pb_enc_fixed32(pb_ostream_t *stream, const pb_field_t *field, const void *src);
+static bool checkreturn pb_enc_fixed64(pb_ostream_t *stream, const pb_field_t *field, const void *src);
+static bool checkreturn pb_enc_bytes(pb_ostream_t *stream, const pb_field_t *field, const void *src);
+static bool checkreturn pb_enc_string(pb_ostream_t *stream, const pb_field_t *field, const void *src);
+static bool checkreturn pb_enc_submessage(pb_ostream_t *stream, const pb_field_t *field, const void *src);
+
 /* --- Function pointers to field encoders ---
  * Order in the array must match pb_action_t LTYPE numbering.
  */
 static const pb_encoder_t PB_ENCODERS[PB_LTYPES_COUNT] = {
     &pb_enc_varint,
+    &pb_enc_uvarint,
     &pb_enc_svarint,
     &pb_enc_fixed32,
     &pb_enc_fixed64,
@@ -32,7 +52,9 @@ static const pb_encoder_t PB_ENCODERS[PB_LTYPES_COUNT] = {
     NULL /* extensions */
 };
 
-/* pb_ostream_t implementation */
+/*******************************
+ * pb_ostream_t implementation *
+ *******************************/
 
 static bool checkreturn buf_write(pb_ostream_t *stream, const uint8_t *buf, size_t count)
 {
@@ -49,7 +71,7 @@ pb_ostream_t pb_ostream_from_buffer(uint8_t *buf, size_t bufsize)
 {
     pb_ostream_t stream;
 #ifdef PB_BUFFER_ONLY
-    stream.callback = (void*)1; /* Just some marker value */
+    stream.callback = (void*)1; /* Just a marker value */
 #else
     stream.callback = &buf_write;
 #endif
@@ -82,7 +104,9 @@ bool checkreturn pb_write(pb_ostream_t *stream, const uint8_t *buf, size_t count
     return true;
 }
 
-/* Main encoding stuff */
+/*************************
+ * Encode a single field *
+ *************************/
 
 /* Encode a static array. Handles the size calculations and possible packing. */
 static bool checkreturn encode_array(pb_ostream_t *stream, const pb_field_t *field,
@@ -94,6 +118,9 @@ static bool checkreturn encode_array(pb_ostream_t *stream, const pb_field_t *fie
     
     if (count == 0)
         return true;
+
+    if (PB_ATYPE(field->type) != PB_ATYPE_POINTER && count > field->array_size)
+        PB_RETURN_ERROR(stream, "array max size exceeded");
     
     /* We always pack arrays if the datatype allows it. */
     if (PB_LTYPE(field->type) <= PB_LTYPE_LAST_PACKABLE)
@@ -145,8 +172,23 @@ static bool checkreturn encode_array(pb_ostream_t *stream, const pb_field_t *fie
         {
             if (!pb_encode_tag_for_field(stream, field))
                 return false;
-            if (!func(stream, field, p))
-                return false;
+
+            /* Normally the data is stored directly in the array entries, but
+             * for pointer-type string and bytes fields, the array entries are
+             * actually pointers themselves also. So we have to dereference once
+             * more to get to the actual data. */
+            if (PB_ATYPE(field->type) == PB_ATYPE_POINTER &&
+                (PB_LTYPE(field->type) == PB_LTYPE_STRING ||
+                 PB_LTYPE(field->type) == PB_LTYPE_BYTES))
+            {
+                if (!func(stream, field, *(const void* const*)p))
+                    return false;      
+            }
+            else
+            {
+                if (!func(stream, field, p))
+                    return false;
+            }
             p = (const char*)p + field->data_size;
         }
     }
@@ -154,25 +196,38 @@ static bool checkreturn encode_array(pb_ostream_t *stream, const pb_field_t *fie
     return true;
 }
 
-/* Encode a field with static allocation, i.e. one whose data is stored
- * in the structure itself. */
-static bool checkreturn encode_static_field(pb_ostream_t *stream,
+/* Encode a field with static or pointer allocation, i.e. one whose data
+ * is available to the encoder directly. */
+static bool checkreturn encode_basic_field(pb_ostream_t *stream,
     const pb_field_t *field, const void *pData)
 {
     pb_encoder_t func;
     const void *pSize;
-    bool dummy = true;
+    bool implicit_has = true;
     
     func = PB_ENCODERS[PB_LTYPE(field->type)];
     
     if (field->size_offset)
         pSize = (const char*)pData + field->size_offset;
     else
-        pSize = &dummy;
-    
+        pSize = &implicit_has;
+
+    if (PB_ATYPE(field->type) == PB_ATYPE_POINTER)
+    {
+        /* pData is a pointer to the field, which contains pointer to
+         * the data. If the 2nd pointer is NULL, it is interpreted as if
+         * the has_field was false.
+         */
+        
+        pData = *(const void* const*)pData;
+        implicit_has = (pData != NULL);
+    }
+
     switch (PB_HTYPE(field->type))
     {
         case PB_HTYPE_REQUIRED:
+            if (!pData)
+                PB_RETURN_ERROR(stream, "missing required field");
             if (!pb_encode_tag_for_field(stream, field))
                 return false;
             if (!func(stream, field, pData))
@@ -191,7 +246,7 @@ static bool checkreturn encode_static_field(pb_ostream_t *stream,
             break;
         
         case PB_HTYPE_REPEATED:
-            if (!encode_array(stream, field, pData, *(const size_t*)pSize, func))
+            if (!encode_array(stream, field, pData, *(const pb_size_t*)pSize, func))
                 return false;
             break;
         
@@ -230,7 +285,8 @@ static bool checkreturn encode_field(pb_ostream_t *stream,
     switch (PB_ATYPE(field->type))
     {
         case PB_ATYPE_STATIC:
-            return encode_static_field(stream, field, pData);
+        case PB_ATYPE_POINTER:
+            return encode_basic_field(stream, field, pData);
         
         case PB_ATYPE_CALLBACK:
             return encode_callback_field(stream, field, pData);
@@ -242,7 +298,7 @@ static bool checkreturn encode_field(pb_ostream_t *stream,
 
 /* Default handler for extension fields. Expects to have a pb_field_t
  * pointer in the extension->type->arg field. */
-static bool checkreturn default_extension_handler(pb_ostream_t *stream,
+static bool checkreturn default_extension_encoder(pb_ostream_t *stream,
     const pb_extension_t *extension)
 {
     const pb_field_t *field = (const pb_field_t*)extension->type->arg;
@@ -255,7 +311,7 @@ static bool checkreturn encode_extension_field(pb_ostream_t *stream,
     const pb_field_t *field, const void *pData)
 {
     const pb_extension_t *extension = *(const pb_extension_t* const *)pData;
-    UNUSED(field);
+    PB_UNUSED(field);
     
     while (extension)
     {
@@ -263,7 +319,7 @@ static bool checkreturn encode_extension_field(pb_ostream_t *stream,
         if (extension->type->encode)
             status = extension->type->encode(stream, extension);
         else
-            status = default_extension_handler(stream, extension);
+            status = default_extension_encoder(stream, extension);
 
         if (!status)
             return false;
@@ -274,39 +330,42 @@ static bool checkreturn encode_extension_field(pb_ostream_t *stream,
     return true;
 }
 
+/*********************
+ * Encode all fields *
+ *********************/
+
+static void *remove_const(const void *p)
+{
+    /* Note: this casts away const, in order to use the common field iterator
+     * logic for both encoding and decoding. */
+    union {
+        void *p1;
+        const void *p2;
+    } t;
+    t.p2 = p;
+    return t.p1;
+}
+
 bool checkreturn pb_encode(pb_ostream_t *stream, const pb_field_t fields[], const void *src_struct)
 {
-    const pb_field_t *field = fields;
-    const void *pData = src_struct;
-    size_t prev_size = 0;
+    pb_field_iter_t iter;
+    if (!pb_field_iter_begin(&iter, fields, remove_const(src_struct)))
+        return true; /* Empty message type */
     
-    while (field->tag != 0)
-    {
-        pData = (const char*)pData + prev_size + field->data_offset;
-        prev_size = field->data_size;
-        
-        /* Special case for static arrays */
-        if (PB_ATYPE(field->type) == PB_ATYPE_STATIC &&
-            PB_HTYPE(field->type) == PB_HTYPE_REPEATED)
-        {
-            prev_size *= field->array_size;
-        }
-        
-        if (PB_LTYPE(field->type) == PB_LTYPE_EXTENSION)
+    do {
+        if (PB_LTYPE(iter.pos->type) == PB_LTYPE_EXTENSION)
         {
             /* Special case for the extension field placeholder */
-            if (!encode_extension_field(stream, field, pData))
+            if (!encode_extension_field(stream, iter.pos, iter.pData))
                 return false;
         }
         else
         {
             /* Regular field */
-            if (!encode_field(stream, field, pData))
+            if (!encode_field(stream, iter.pos, iter.pData))
                 return false;
         }
-    
-        field++;
-    }
+    } while (pb_field_iter_next(&iter));
     
     return true;
 }
@@ -316,7 +375,20 @@ bool pb_encode_delimited(pb_ostream_t *stream, const pb_field_t fields[], const
     return pb_encode_submessage(stream, fields, src_struct);
 }
 
-/* Helper functions */
+bool pb_get_encoded_size(size_t *size, const pb_field_t fields[], const void *src_struct)
+{
+    pb_ostream_t stream = PB_OSTREAM_SIZING;
+    
+    if (!pb_encode(&stream, fields, src_struct))
+        return false;
+    
+    *size = stream.bytes_written;
+    return true;
+}
+
+/********************
+ * Helper functions *
+ ********************/
 bool checkreturn pb_encode_varint(pb_ostream_t *stream, uint64_t value)
 {
     uint8_t buffer[10];
@@ -340,9 +412,9 @@ bool checkreturn pb_encode_svarint(pb_ostream_t *stream, int64_t value)
 {
     uint64_t zigzagged;
     if (value < 0)
-        zigzagged = (uint64_t)(~(value << 1));
+        zigzagged = ~((uint64_t)value << 1);
     else
-        zigzagged = (uint64_t)(value << 1);
+        zigzagged = (uint64_t)value << 1;
     
     return pb_encode_varint(stream, zigzagged);
 }
@@ -383,7 +455,7 @@ bool checkreturn pb_encode_fixed64(pb_ostream_t *stream, const void *value)
 
 bool checkreturn pb_encode_tag(pb_ostream_t *stream, pb_wire_type_t wiretype, uint32_t field_number)
 {
-    uint64_t tag = wiretype | (field_number << 3);
+    uint64_t tag = ((uint64_t)field_number << 3) | wiretype;
     return pb_encode_varint(stream, tag);
 }
 
@@ -393,6 +465,7 @@ bool checkreturn pb_encode_tag_for_field(pb_ostream_t *stream, const pb_field_t
     switch (PB_LTYPE(field->type))
     {
         case PB_LTYPE_VARINT:
+        case PB_LTYPE_UVARINT:
         case PB_LTYPE_SVARINT:
             wiretype = PB_WT_VARINT;
             break;
@@ -434,7 +507,12 @@ bool checkreturn pb_encode_submessage(pb_ostream_t *stream, const pb_field_t fie
     bool status;
     
     if (!pb_encode(&substream, fields, src_struct))
+    {
+#ifndef PB_NO_ERRMSG
+        stream->errmsg = substream.errmsg;
+#endif
         return false;
+    }
     
     size = substream.bytes_written;
     
@@ -473,14 +551,30 @@ bool checkreturn pb_encode_submessage(pb_ostream_t *stream, const pb_field_t fie
 
 /* Field encoders */
 
-bool checkreturn pb_enc_varint(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_varint(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+{
+    int64_t value = 0;
+    
+    /* Cases 1 and 2 are for compilers that have smaller types for bool
+     * or enums. */
+    switch (field->data_size)
+    {
+        case 1: value = *(const int8_t*)src; break;
+        case 2: value = *(const int16_t*)src; break;
+        case 4: value = *(const int32_t*)src; break;
+        case 8: value = *(const int64_t*)src; break;
+        default: PB_RETURN_ERROR(stream, "invalid data_size");
+    }
+    
+    return pb_encode_varint(stream, (uint64_t)value);
+}
+
+static bool checkreturn pb_enc_uvarint(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
     uint64_t value = 0;
     
     switch (field->data_size)
     {
-        case 1: value = *(const uint8_t*)src; break;
-        case 2: value = *(const uint16_t*)src; break;
         case 4: value = *(const uint32_t*)src; break;
         case 8: value = *(const uint64_t*)src; break;
         default: PB_RETURN_ERROR(stream, "invalid data_size");
@@ -489,7 +583,7 @@ bool checkreturn pb_enc_varint(pb_ostream_t *stream, const pb_field_t *field, co
     return pb_encode_varint(stream, value);
 }
 
-bool checkreturn pb_enc_svarint(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_svarint(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
     int64_t value = 0;
     
@@ -503,40 +597,64 @@ bool checkreturn pb_enc_svarint(pb_ostream_t *stream, const pb_field_t *field, c
     return pb_encode_svarint(stream, value);
 }
 
-bool checkreturn pb_enc_fixed64(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_fixed64(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
-    UNUSED(field);
+    PB_UNUSED(field);
     return pb_encode_fixed64(stream, src);
 }
 
-bool checkreturn pb_enc_fixed32(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_fixed32(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
-    UNUSED(field);
+    PB_UNUSED(field);
     return pb_encode_fixed32(stream, src);
 }
 
-bool checkreturn pb_enc_bytes(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_bytes(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
     const pb_bytes_array_t *bytes = (const pb_bytes_array_t*)src;
-    UNUSED(field);
+    
+    if (src == NULL)
+    {
+        /* Threat null pointer as an empty bytes field */
+        return pb_encode_string(stream, NULL, 0);
+    }
+    
+    if (PB_ATYPE(field->type) == PB_ATYPE_STATIC &&
+        PB_BYTES_ARRAY_T_ALLOCSIZE(bytes->size) > field->data_size)
+    {
+        PB_RETURN_ERROR(stream, "bytes size exceeded");
+    }
+    
     return pb_encode_string(stream, bytes->bytes, bytes->size);
 }
 
-bool checkreturn pb_enc_string(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_string(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
-    /* strnlen() is not always available, so just use a for-loop */
     size_t size = 0;
+    size_t max_size = field->data_size;
     const char *p = (const char*)src;
-    while (size < field->data_size && *p != '\0')
+    
+    if (PB_ATYPE(field->type) == PB_ATYPE_POINTER)
+        max_size = (size_t)-1;
+
+    if (src == NULL)
     {
-        size++;
-        p++;
+        size = 0; /* Threat null pointer as an empty string */
+    }
+    else
+    {
+        /* strnlen() is not always available, so just use a loop */
+        while (size < max_size && *p != '\0')
+        {
+            size++;
+            p++;
+        }
     }
 
     return pb_encode_string(stream, (const uint8_t*)src, size);
 }
 
-bool checkreturn pb_enc_submessage(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_submessage(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
     if (field->ptr == NULL)
         PB_RETURN_ERROR(stream, "invalid field descriptor");

pb_encode.h

@@ -3,8 +3,8 @@
  * field descriptions created by nanopb_generator.py.
  */
 
-#ifndef _PB_ENCODE_H_
-#define _PB_ENCODE_H_
+#ifndef PB_ENCODE_H_INCLUDED
+#define PB_ENCODE_H_INCLUDED
 
 #include "pb.h"
 
@@ -24,7 +24,7 @@ extern "C" {
  * 4) Substreams will modify max_size and bytes_written. Don't use them
  *    to calculate any pointers.
  */
-struct _pb_ostream_t
+struct pb_ostream_s
 {
 #ifdef PB_BUFFER_ONLY
     /* Callback pointer is not used in buffer-only configuration.
@@ -71,6 +71,10 @@ bool pb_encode(pb_ostream_t *stream, const pb_field_t fields[], const void *src_
  */
 bool pb_encode_delimited(pb_ostream_t *stream, const pb_field_t fields[], const void *src_struct);
 
+/* Encode the message to get the size of the encoded data, but do not store
+ * the data. */
+bool pb_get_encoded_size(size_t *size, const pb_field_t fields[], const void *src_struct);
+
 /**************************************
  * Functions for manipulating streams *
  **************************************/
@@ -143,24 +147,6 @@ bool pb_encode_fixed64(pb_ostream_t *stream, const void *value);
  */
 bool pb_encode_submessage(pb_ostream_t *stream, const pb_field_t fields[], const void *src_struct);
 
-
-/*******************************
- * Internal / legacy functions *
- *******************************/
-
-#ifdef NANOPB_INTERNALS
-bool pb_enc_varint(pb_ostream_t *stream, const pb_field_t *field, const void *src);
-bool pb_enc_svarint(pb_ostream_t *stream, const pb_field_t *field, const void *src);
-bool pb_enc_fixed32(pb_ostream_t *stream, const pb_field_t *field, const void *src);
-bool pb_enc_fixed64(pb_ostream_t *stream, const pb_field_t *field, const void *src);
-bool pb_enc_bytes(pb_ostream_t *stream, const pb_field_t *field, const void *src);
-bool pb_enc_string(pb_ostream_t *stream, const pb_field_t *field, const void *src);
-#endif
-
-/* This function is not recommended for new programs. Use pb_encode_submessage()
- * instead, it has the same functionality with a less confusing interface. */
-bool pb_enc_submessage(pb_ostream_t *stream, const pb_field_t *field, const void *src);
-
 #ifdef __cplusplus
 } /* extern "C" */
 #endif

tests/Makefile

@@ -1,143 +1,21 @@
-CFLAGS=-ansi -Wall -Werror -I .. -g -O0
-DEPS=../pb_decode.h ../pb_encode.h ../pb.h person.pb.h \
-	callbacks2.pb.h callbacks.pb.h unittests.h unittestproto.pb.h \
-	alltypes.pb.h missing_fields.pb.h
-TESTS=  decode_unittests encode_unittests \
-	test_decode1 test_decode2 test_decode3 test_decode3_buf \
-	test_encode1 test_encode2 test_encode3 test_encode3_buf \
-	test_decode_callbacks test_encode_callbacks \
-	test_missing_fields test_no_messages test_funny_name \
-	test_multiple_files test_cxxcompile test_options \
-	bc_encode bc_decode test_encode_extensions test_decode_extensions
-
-# More strict checks for the core part of nanopb
-CC_VERSION=$(shell $(CC) -v 2>&1)
-CFLAGS_CORE=
-ifneq (,$(findstring gcc,$(CC_VERSION)))
-	CFLAGS_CORE=-pedantic -Wextra -Wcast-qual -Wlogical-op -Wconversion
-	CFLAGS+=--coverage -fstack-protector-all
-	LDFLAGS+=--coverage
-endif
-ifneq (,$(findstring clang,$(CC_VERSION)))
-	CFLAGS_CORE=-pedantic -Wextra -Wcast-qual -Wconversion
-endif
-
-# Also use mudflap if it is available
-# To enable, run with    make -B USE_MUDFLAP=y
-USE_MUDFLAP ?= n
-ifeq ($(USE_MUDFLAP),y)
-	CFLAGS += -fmudflap
-	LDFLAGS += -lmudflap -fmudflap
-endif
-
-all: breakpoints $(TESTS) run_unittests
+all:
+	scons
 
 clean:
-	rm -f $(TESTS) person.pb* alltypes.pb* *.o *.gcda *.gcno *.pb.h *.pb.c
+	scons -c
 
-%.pb.o: %.pb.c %.pb.h
-	$(CC) $(CFLAGS) $(CFLAGS_CORE) -c -o $@ $<
+coverage:
+	rm -rf build coverage
 
-%.o: %.c
-%.o: %.c $(DEPS)
-	$(CC) $(CFLAGS) -c -o $@ $<
+	# LCOV does not like the newer gcov format
+	scons CC=gcc-4.6 CXX=gcc-4.6
 
-pb_encode.o: ../pb_encode.c $(DEPS)
-	$(CC) $(CFLAGS) $(CFLAGS_CORE) -c -o $@ $<
-pb_decode.o: ../pb_decode.c $(DEPS)
-	$(CC) $(CFLAGS) $(CFLAGS_CORE) -c -o $@ $<
+	# Collect the data
+	mkdir build/coverage
+	lcov --base-directory . --directory build/ --gcov-tool gcov-4.6 -c -o build/coverage/nanopb.info
 
-# Test for compilability with c++ compiler
+	# Remove the test code from results
+	lcov -r build/coverage/nanopb.info '*tests*' -o build/coverage/nanopb.info
 
-pb_encode.cxx.o: ../pb_encode.c $(DEPS)
-	$(CXX) $(CFLAGS) $(CFLAGS_CORE) -c -o $@ $<
-pb_decode.cxx.o: ../pb_decode.c $(DEPS)
-	$(CXX) $(CFLAGS) $(CFLAGS_CORE) -c -o $@ $<
-
-# Test for PB_BUF_ONLY compilation option
-
-pb_encode.buf.o: ../pb_encode.c $(DEPS)
-	$(CC) -DPB_BUFFER_ONLY $(CFLAGS) $(CFLAGS_CORE) -c -o $@ $<
-pb_decode.buf.o: ../pb_decode.c $(DEPS)
-	$(CC) -DPB_BUFFER_ONLY $(CFLAGS) $(CFLAGS_CORE) -c -o $@ $<
-%.buf.o: %.c $(DEPS)
-	$(CC) -DPB_BUFFER_ONLY $(CFLAGS) -c -o $@ $<	
-test_encode3_buf: test_encode3.buf.o pb_encode.buf.o alltypes.pb.o
-	$(CC) $(LDFLAGS) $^ -o $@
-test_decode3_buf: test_decode3.buf.o pb_decode.buf.o alltypes.pb.o
-	$(CC) $(LDFLAGS) $^ -o $@
-
-test_cxxcompile: pb_encode.cxx.o pb_decode.cxx.o
-test_decode1: test_decode1.o pb_decode.o person.pb.o
-test_decode2: test_decode2.o pb_decode.o person.pb.o
-test_decode3: test_decode3.o pb_decode.o alltypes.pb.o
-test_encode1: test_encode1.o pb_encode.o person.pb.o
-test_encode2: test_encode2.o pb_encode.o person.pb.o
-test_encode3: test_encode3.o pb_encode.o alltypes.pb.o
-test_multiple_files: test_multiple_files.o pb_encode.o callbacks2.pb.o callbacks.pb.o
-test_decode_callbacks: test_decode_callbacks.o pb_decode.o callbacks.pb.o
-test_encode_callbacks: test_encode_callbacks.o pb_encode.o callbacks.pb.o
-test_missing_fields: test_missing_fields.o pb_encode.o pb_decode.o missing_fields.pb.o
-decode_unittests: decode_unittests.o pb_decode.o unittestproto.pb.o
-encode_unittests: encode_unittests.o pb_encode.o unittestproto.pb.o
-test_no_messages: no_messages.pb.h no_messages.pb.c no_messages.pb.o
-test_funny_name: funny-proto+name.pb.h funny-proto+name.pb.o
-bc_encode: bc_alltypes.pb.o pb_encode.o bc_encode.o
-bc_decode: bc_alltypes.pb.o pb_decode.o bc_decode.o
-test_encode_extensions: test_encode_extensions.c pb_encode.o alltypes.pb.o extensions.pb.o
-test_decode_extensions: test_decode_extensions.c pb_decode.o alltypes.pb.o extensions.pb.o
-
-%.pb: %.proto
-	protoc -I. -I../generator -I/usr/include -o$@ $<
-
-%.pb.c %.pb.h: %.pb ../generator/nanopb_generator.py
-	python ../generator/nanopb_generator.py $<
-
-breakpoints: ../*.c *.c
-	grep -n 'return false;' $^ | cut -d: -f-2 | xargs -n 1 echo b > $@
-
-coverage: run_unittests
-	gcov pb_encode.gcda
-	gcov pb_decode.gcda
-
-run_unittests: $(TESTS)
-	rm -f *.gcda
-	
-	./decode_unittests > /dev/null
-	./encode_unittests > /dev/null
-	
-	[ "`./test_encode1 | ./test_decode1`" = \
-	"`./test_encode1 | protoc --decode=Person -I. -I../generator -I/usr/include person.proto`" ]
-
-	[ "`./test_encode2 | ./test_decode1`" = \
-	"`./test_encode2 | protoc --decode=Person -I. -I../generator -I/usr/include person.proto`" ]
-
-	[ "`./test_encode2 | ./test_decode2`" = \
-	"`./test_encode2 | protoc --decode=Person -I. -I../generator -I/usr/include person.proto`" ]
-	
-	[ "`./test_decode2 < person_with_extra_field.pb`" = \
-	"`./test_encode2 | ./test_decode2`" ]
-	
-	[ "`./test_encode_callbacks | ./test_decode_callbacks`" = \
-	"`./test_encode_callbacks | protoc --decode=TestMessage callbacks.proto`" ]
-
-	./test_encode3 | ./test_decode3
-	./test_encode3 1 | ./test_decode3 1
-	./test_encode3 1 | protoc --decode=AllTypes -I. -I../generator -I/usr/include alltypes.proto >/dev/null
-	./test_encode3_buf 1 | ./test_decode3_buf 1
-	./test_decode3 < alltypes_with_extra_fields.pb
-	./bc_encode | ./bc_decode
-	./test_encode_extensions | ./test_decode_extensions
-
-	./test_missing_fields
-
-test_options: options.pb.h options.expected options.pb.o
-	cat options.expected | while read -r p; do \
-	    if ! grep -q "$$p" $<; then \
-	        echo Expected: "$$p"; \
-	        exit 1; \
-	    fi \
-	done
-
-run_fuzztest: test_decode3
-	bash -c 'ulimit -c unlimited; I=1; while true; do cat /dev/urandom | ./test_decode3 > /dev/null; I=$$(($$I+1)); echo -en "\r$$I"; done'
+	# Generate HTML
+	genhtml -o build/coverage build/coverage/nanopb.info

tests/SConstruct

@@ -0,0 +1,147 @@
+Help('''
+Type 'scons' to build and run all the available test cases.
+It will automatically detect your platform and C compiler and
+build appropriately.
+
+You can modify the behavious using following options:
+CC          Name of C compiler
+CXX         Name of C++ compiler
+CCFLAGS     Flags to pass to the C compiler
+CXXFLAGS    Flags to pass to the C++ compiler
+
+For example, for a clang build, use:
+scons CC=clang CXX=clang++
+''')
+
+import os
+env = Environment(ENV = os.environ, tools = ['default', 'nanopb'])
+
+# Allow overriding the compiler with scons CC=???
+if 'CC' in ARGUMENTS: env.Replace(CC = ARGUMENTS['CC'])
+if 'CXX' in ARGUMENTS: env.Replace(CXX = ARGUMENTS['CXX'])
+if 'CCFLAGS' in ARGUMENTS: env.Append(CCFLAGS = ARGUMENTS['CCFLAGS'])
+if 'CXXFLAGS' in ARGUMENTS: env.Append(CXXFLAGS = ARGUMENTS['CXXFLAGS'])
+
+# Add the builders defined in site_init.py
+add_nanopb_builders(env)
+
+# Path to the files shared by tests, and to the nanopb core.
+env.Append(CPPPATH = ["#../", "$COMMON"])
+
+# Path for finding nanopb.proto
+env.Append(PROTOCPATH = '#../generator')
+
+# Check the compilation environment, unless we are just cleaning up.
+if not env.GetOption('clean'):
+    def check_ccflags(context, flags, linkflags = ''):
+        '''Check if given CCFLAGS are supported'''
+        context.Message('Checking support for CCFLAGS="%s"... ' % flags)
+        oldflags = context.env['CCFLAGS']
+        oldlinkflags = context.env['CCFLAGS']
+        context.env.Append(CCFLAGS = flags)
+        context.env.Append(LINKFLAGS = linkflags)
+        result = context.TryCompile("int main() {return 0;}", '.c')
+        context.env.Replace(CCFLAGS = oldflags)
+        context.env.Replace(LINKFLAGS = oldlinkflags)
+        context.Result(result)
+        return result
+    
+    conf = Configure(env, custom_tests = {'CheckCCFLAGS': check_ccflags})
+
+    # If the platform doesn't support C99, use our own header file instead.
+    stdbool = conf.CheckCHeader('stdbool.h')
+    stdint = conf.CheckCHeader('stdint.h')
+    stddef = conf.CheckCHeader('stddef.h')
+    string = conf.CheckCHeader('string.h')
+    stdlib = conf.CheckCHeader('stdlib.h')
+    if not stdbool or not stdint or not stddef or not string:
+        conf.env.Append(CPPDEFINES = {'PB_SYSTEM_HEADER': '\\"pb_syshdr.h\\"'})
+        conf.env.Append(CPPPATH = "#../extra")
+	conf.env.Append(SYSHDR = '\\"pb_syshdr.h\\"')
+        
+        if stdbool: conf.env.Append(CPPDEFINES = {'HAVE_STDBOOL_H': 1})
+        if stdint: conf.env.Append(CPPDEFINES = {'HAVE_STDINT_H': 1})
+        if stddef: conf.env.Append(CPPDEFINES = {'HAVE_STDDEF_H': 1})
+        if string: conf.env.Append(CPPDEFINES = {'HAVE_STRING_H': 1})
+        if stdlib: conf.env.Append(CPPDEFINES = {'HAVE_STDLIB_H': 1})
+    
+    # Check if we can use pkg-config to find protobuf include path
+    status, output = conf.TryAction('pkg-config protobuf --variable=includedir > $TARGET')
+    if status:
+        conf.env.Append(PROTOCPATH = output.strip())
+    else:
+        conf.env.Append(PROTOCPATH = '/usr/include')
+    
+    # Check if libmudflap is available (only with GCC)
+    if 'gcc' in env['CC']:
+        if conf.CheckLib('mudflap'):
+            conf.env.Append(CCFLAGS = '-fmudflap')
+            conf.env.Append(LINKFLAGS = '-fmudflap')
+    
+    # Check if we can use extra strict warning flags (only with GCC)
+    extra = '-Wcast-qual -Wlogical-op -Wconversion'
+    extra += ' -fstrict-aliasing -Wstrict-aliasing=1'
+    extra += ' -Wmissing-prototypes -Wmissing-declarations -Wredundant-decls'
+    extra += ' -Wstack-protector '
+    if 'gcc' in env['CC']:
+        if conf.CheckCCFLAGS(extra):
+            conf.env.Append(CORECFLAGS = extra)
+    
+    # Check if we can use undefined behaviour sanitizer (only with clang)
+    extra = '-fsanitize=undefined '
+    if 'clang' in env['CC']:
+        if conf.CheckCCFLAGS(extra, linkflags = extra):
+            conf.env.Append(CORECFLAGS = extra)
+            conf.env.Append(LINKFLAGS = extra)
+    
+    # End the config stuff
+    env = conf.Finish()
+
+# Initialize the CCFLAGS according to the compiler
+if 'gcc' in env['CC']:
+    # GNU Compiler Collection
+    
+    # Debug info, warnings as errors
+    env.Append(CFLAGS = '-ansi -pedantic -g -Wall -Werror -fprofile-arcs -ftest-coverage ')
+    env.Append(CORECFLAGS = '-Wextra')
+    env.Append(LINKFLAGS = '-g --coverage')
+    
+    # We currently need uint64_t anyway, even though ANSI C90 otherwise..
+    env.Append(CFLAGS = '-Wno-long-long')
+elif 'clang' in env['CC']:
+    # CLang
+    env.Append(CFLAGS = '-ansi -g -Wall -Werror')
+    env.Append(CORECFLAGS = ' -Wextra -Wcast-qual -Wconversion')
+elif 'cl' in env['CC']:
+    # Microsoft Visual C++
+    
+    # Debug info on, warning level 2 for tests, warnings as errors
+    env.Append(CFLAGS = '/Zi /W2 /WX')
+    env.Append(LINKFLAGS = '/DEBUG')
+    
+    # More strict checks on the nanopb core
+    env.Append(CORECFLAGS = '/W4')
+    
+    # PB_RETURN_ERROR triggers C4127 because of while(0)
+    env.Append(CFLAGS = '/wd4127')
+elif 'tcc' in env['CC']:
+    # Tiny C Compiler
+    env.Append(CFLAGS = '-Wall -Werror -g')
+
+env.SetDefault(CORECFLAGS = '')
+
+if 'clang' in env['CXX']:
+    env.Append(CXXFLAGS = '-g -Wall -Werror -Wextra -Wno-missing-field-initializers')
+elif 'g++' in env['CXX'] or 'gcc' in env['CXX']:
+    env.Append(CXXFLAGS = '-g -Wall -Werror -Wextra -Wno-missing-field-initializers')
+elif 'cl' in env['CXX']:
+    env.Append(CXXFLAGS = '/Zi /W2 /WX')
+    
+# Now include the SConscript files from all subdirectories
+import os.path
+env['VARIANT_DIR'] = 'build'
+env['BUILD'] = '#' + env['VARIANT_DIR']
+env['COMMON'] = '#' + env['VARIANT_DIR'] + '/common'
+for subdir in Glob('*/SConscript') + Glob('regression/*/SConscript'):
+    SConscript(subdir, exports = 'env', variant_dir = env['VARIANT_DIR'] + '/' + os.path.dirname(str(subdir)))
+

tests/alltypes/SConscript

@@ -0,0 +1,35 @@
+# Build and run a test that encodes and decodes a message that contains
+# all of the Protocol Buffers data types.
+
+Import("env")
+
+env.NanopbProto(["alltypes", "alltypes.options"])
+enc = env.Program(["encode_alltypes.c", "alltypes.pb.c", "$COMMON/pb_encode.o", "$COMMON/pb_common.o"])
+dec = env.Program(["decode_alltypes.c", "alltypes.pb.c", "$COMMON/pb_decode.o", "$COMMON/pb_common.o"])
+
+# Test the round-trip from nanopb encoder to nanopb decoder
+env.RunTest(enc)
+env.RunTest([dec, "encode_alltypes.output"])
+
+# Re-encode the data using protoc, and check that the results from nanopb
+# match byte-per-byte to the protoc output.
+env.Decode("encode_alltypes.output.decoded",
+           ["encode_alltypes.output", "alltypes.proto"],
+           MESSAGE='AllTypes')
+env.Encode("encode_alltypes.output.recoded",
+           ["encode_alltypes.output.decoded", "alltypes.proto"],
+           MESSAGE='AllTypes')
+env.Compare(["encode_alltypes.output", "encode_alltypes.output.recoded"])
+
+# Do the same checks with the optional fields present.
+env.RunTest("optionals.output", enc, ARGS = ['1'])
+env.RunTest("optionals.decout", [dec, "optionals.output"], ARGS = ['1'])
+env.Decode("optionals.output.decoded",
+           ["optionals.output", "alltypes.proto"],
+           MESSAGE='AllTypes')
+env.Encode("optionals.output.recoded",
+           ["optionals.output.decoded", "alltypes.proto"],
+           MESSAGE='AllTypes')
+env.Compare(["optionals.output", "optionals.output.recoded"])
+
+

tests/alltypes/alltypes.proto

@@ -0,0 +1,114 @@
+message SubMessage {
+    required string substuff1 = 1 [default = "1"];
+    required int32 substuff2 = 2 [default = 2];
+    optional fixed32 substuff3 = 3 [default = 3];
+}
+
+message EmptyMessage {
+
+}
+
+enum HugeEnum {
+    Negative = -2147483647; /* protoc doesn't accept -2147483648 here */
+    Positive =  2147483647;
+}
+
+message Limits {
+    required int32      int32_min  =  1 [default = 2147483647];
+    required int32      int32_max  =  2 [default = -2147483647];
+    required uint32     uint32_min =  3 [default = 4294967295];
+    required uint32     uint32_max =  4 [default = 0];
+    required int64      int64_min  =  5 [default = 9223372036854775807];
+    required int64      int64_max  =  6 [default = -9223372036854775807];
+    required uint64     uint64_min =  7 [default = 18446744073709551615];
+    required uint64     uint64_max =  8 [default = 0];
+    required HugeEnum   enum_min   =  9 [default = Positive];
+    required HugeEnum   enum_max   = 10 [default = Negative];
+}
+
+enum MyEnum {
+    Zero = 0;
+    First = 1;
+    Second = 2;
+    Truth = 42;
+}
+
+message AllTypes {
+    required int32      req_int32   = 1;
+    required int64      req_int64   = 2;
+    required uint32     req_uint32  = 3;
+    required uint64     req_uint64  = 4;
+    required sint32     req_sint32  = 5;
+    required sint64     req_sint64  = 6;
+    required bool       req_bool    = 7;
+    
+    required fixed32    req_fixed32 = 8;
+    required sfixed32   req_sfixed32= 9;
+    required float      req_float   = 10;
+    
+    required fixed64    req_fixed64 = 11;
+    required sfixed64   req_sfixed64= 12;
+    required double     req_double  = 13;
+    
+    required string     req_string  = 14;
+    required bytes      req_bytes   = 15;
+    required SubMessage req_submsg  = 16;
+    required MyEnum     req_enum    = 17;
+    required EmptyMessage req_emptymsg = 18;
+    
+    
+    repeated int32      rep_int32   = 21 [packed = true];
+    repeated int64      rep_int64   = 22 [packed = true];
+    repeated uint32     rep_uint32  = 23 [packed = true];
+    repeated uint64     rep_uint64  = 24 [packed = true];
+    repeated sint32     rep_sint32  = 25 [packed = true];
+    repeated sint64     rep_sint64  = 26 [packed = true];
+    repeated bool       rep_bool    = 27 [packed = true];
+    
+    repeated fixed32    rep_fixed32 = 28 [packed = true];
+    repeated sfixed32   rep_sfixed32= 29 [packed = true];
+    repeated float      rep_float   = 30 [packed = true];
+    
+    repeated fixed64    rep_fixed64 = 31 [packed = true];
+    repeated sfixed64   rep_sfixed64= 32 [packed = true];
+    repeated double     rep_double  = 33 [packed = true];
+    
+    repeated string     rep_string  = 34;
+    repeated bytes      rep_bytes   = 35;
+    repeated SubMessage rep_submsg  = 36;
+    repeated MyEnum     rep_enum    = 37 [packed = true];
+    repeated EmptyMessage rep_emptymsg = 38;
+    
+    optional int32      opt_int32   = 41 [default = 4041];
+    optional int64      opt_int64   = 42 [default = 4042];
+    optional uint32     opt_uint32  = 43 [default = 4043];
+    optional uint64     opt_uint64  = 44 [default = 4044];
+    optional sint32     opt_sint32  = 45 [default = 4045];
+    optional sint64     opt_sint64  = 46 [default = 4046];
+    optional bool       opt_bool    = 47 [default = false];
+    
+    optional fixed32    opt_fixed32 = 48 [default = 4048];
+    optional sfixed32   opt_sfixed32= 49 [default = 4049];
+    optional float      opt_float   = 50 [default = 4050];
+    
+    optional fixed64    opt_fixed64 = 51 [default = 4051];
+    optional sfixed64   opt_sfixed64= 52 [default = 4052];
+    optional double     opt_double  = 53 [default = 4053];
+    
+    optional string     opt_string  = 54 [default = "4054"];
+    optional bytes      opt_bytes   = 55 [default = "4055"];
+    optional SubMessage opt_submsg  = 56;
+    optional MyEnum     opt_enum    = 57 [default = Second];
+    optional EmptyMessage opt_emptymsg = 58;
+
+    // Check that extreme integer values are handled correctly
+    required Limits     req_limits = 98;
+
+    // Just to make sure that the size of the fields has been calculated
+    // properly, i.e. otherwise a bug in last field might not be detected.
+    required int32      end = 99;
+
+
+    extensions 200 to 255;
+}
+

tests/alltypes/decode_alltypes.c

@@ -8,6 +8,7 @@
 #include <stdlib.h>
 #include <pb_decode.h>
 #include "alltypes.pb.h"
+#include "test_helpers.h"
 
 #define TEST(x) if (!(x)) { \
     printf("Test " #x " failed.\n"); \
@@ -18,10 +19,12 @@
    the decoding and checks the fields. */
 bool check_alltypes(pb_istream_t *stream, int mode)
 {
-    AllTypes alltypes;
+    /* Uses _init_default to just make sure that it works. */
+    AllTypes alltypes = AllTypes_init_default;
     
     /* Fill with garbage to better detect initialization errors */
     memset(&alltypes, 0xAA, sizeof(alltypes));
+    alltypes.extensions = 0;
     
     if (!pb_decode(stream, AllTypes_fields, &alltypes))
         return false;
@@ -169,6 +172,17 @@ bool check_alltypes(pb_istream_t *stream, int mode)
         TEST(alltypes.has_opt_emptymsg  == true);
     }
     
+    TEST(alltypes.req_limits.int32_min  == INT32_MIN);
+    TEST(alltypes.req_limits.int32_max  == INT32_MAX);
+    TEST(alltypes.req_limits.uint32_min == 0);
+    TEST(alltypes.req_limits.uint32_max == UINT32_MAX);
+    TEST(alltypes.req_limits.int64_min  == INT64_MIN);
+    TEST(alltypes.req_limits.int64_max  == INT64_MAX);
+    TEST(alltypes.req_limits.uint64_min == 0);
+    TEST(alltypes.req_limits.uint64_max == UINT64_MAX);
+    TEST(alltypes.req_limits.enum_min   == HugeEnum_Negative);
+    TEST(alltypes.req_limits.enum_max   == HugeEnum_Positive);
+    
     TEST(alltypes.end == 1099);
     
     return true;
@@ -176,15 +190,19 @@ bool check_alltypes(pb_istream_t *stream, int mode)
 
 int main(int argc, char **argv)
 {
+    uint8_t buffer[1024];
+    size_t count;
+    pb_istream_t stream;
+
     /* Whether to expect the optional values or the default values. */
     int mode = (argc > 1) ? atoi(argv[1]) : 0;
     
     /* Read the data into buffer */
-    uint8_t buffer[1024];
-    size_t count = fread(buffer, 1, sizeof(buffer), stdin);
+    SET_BINARY_MODE(stdin);
+    count = fread(buffer, 1, sizeof(buffer), stdin);
     
     /* Construct a pb_istream_t for reading from the buffer */
-    pb_istream_t stream = pb_istream_from_buffer(buffer, count);
+    stream = pb_istream_from_buffer(buffer, count);
     
     /* Decode and print out the stuff */
     if (!check_alltypes(&stream, mode))

tests/alltypes/encode_alltypes.c

@@ -6,13 +6,14 @@
 #include <string.h>
 #include <pb_encode.h>
 #include "alltypes.pb.h"
+#include "test_helpers.h"
 
 int main(int argc, char **argv)
 {
     int mode = (argc > 1) ? atoi(argv[1]) : 0;
     
     /* Initialize the structure with constants */
-    AllTypes alltypes = {0};
+    AllTypes alltypes = AllTypes_init_zero;
     
     alltypes.req_int32         = -1001;
     alltypes.req_int64         = -1002;
@@ -66,6 +67,17 @@ int main(int argc, char **argv)
     alltypes.rep_enum_count = 5; alltypes.rep_enum[4] = MyEnum_Truth;
     alltypes.rep_emptymsg_count = 5;
     
+    alltypes.req_limits.int32_min  = INT32_MIN;
+    alltypes.req_limits.int32_max  = INT32_MAX;
+    alltypes.req_limits.uint32_min = 0;
+    alltypes.req_limits.uint32_max = UINT32_MAX;
+    alltypes.req_limits.int64_min  = INT64_MIN;
+    alltypes.req_limits.int64_max  = INT64_MAX;
+    alltypes.req_limits.uint64_min = 0;
+    alltypes.req_limits.uint64_max = UINT64_MAX;
+    alltypes.req_limits.enum_min   = HugeEnum_Negative;
+    alltypes.req_limits.enum_max   = HugeEnum_Positive;
+    
     if (mode != 0)
     {
         /* Fill in values for optional fields */
@@ -113,18 +125,21 @@ int main(int argc, char **argv)
     
     alltypes.end = 1099;
     
-    uint8_t buffer[1024];
-    pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
-    
-    /* Now encode it and check if we succeeded. */
-    if (pb_encode(&stream, AllTypes_fields, &alltypes))
     {
-        fwrite(buffer, 1, stream.bytes_written, stdout);
-        return 0; /* Success */
-    }
-    else
-    {
-        fprintf(stderr, "Encoding failed: %s\n", PB_GET_ERROR(&stream));
-        return 1; /* Failure */
+        uint8_t buffer[AllTypes_size];
+        pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
+        
+        /* Now encode it and check if we succeeded. */
+        if (pb_encode(&stream, AllTypes_fields, &alltypes))
+        {
+            SET_BINARY_MODE(stdout);
+            fwrite(buffer, 1, stream.bytes_written, stdout);
+            return 0; /* Success */
+        }
+        else
+        {
+            fprintf(stderr, "Encoding failed: %s\n", PB_GET_ERROR(&stream));
+            return 1; /* Failure */
+        }
     }
 }

tests/alltypes_callback/SConscript

@@ -0,0 +1,23 @@
+# Test the AllTypes encoding & decoding using callbacks for all fields.
+
+Import("env")
+
+c = Copy("$TARGET", "$SOURCE")
+env.Command("alltypes.proto", "#alltypes/alltypes.proto", c)
+
+env.NanopbProto(["alltypes", "alltypes.options"])
+enc = env.Program(["encode_alltypes_callback.c", "alltypes.pb.c", "$COMMON/pb_encode.o", "$COMMON/pb_common.o"])
+dec = env.Program(["decode_alltypes_callback.c", "alltypes.pb.c", "$COMMON/pb_decode.o", "$COMMON/pb_common.o"])
+
+refdec = "$BUILD/alltypes/decode_alltypes$PROGSUFFIX"
+
+# Encode and compare results
+env.RunTest(enc)
+env.RunTest("decode_alltypes.output", [refdec, "encode_alltypes_callback.output"])
+env.RunTest("decode_alltypes_callback.output", [dec, "encode_alltypes_callback.output"])
+
+# Do the same thing with the optional fields present
+env.RunTest("optionals.output", enc, ARGS = ['1'])
+env.RunTest("optionals.refdecout", [refdec, "optionals.output"], ARGS = ['1'])
+env.RunTest("optionals.decout", [dec, "optionals.output"], ARGS = ['1'])
+

tests/alltypes_callback/alltypes.options

@@ -0,0 +1,3 @@
+# Generate all fields as callbacks.
+AllTypes.*              type:FT_CALLBACK
+SubMessage.substuff1    max_size:16

tests/alltypes_callback/decode_alltypes_callback.c

@@ -0,0 +1,424 @@
+/* Attempts to test all the datatypes supported by ProtoBuf when used as callback fields.
+ * Note that normally there would be no reason to use callback fields for this,
+ * because each encoder defined here only gives a single field.
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdlib.h>
+#include <pb_decode.h>
+#include "alltypes.pb.h"
+#include "test_helpers.h"
+
+#define TEST(x) if (!(x)) { \
+    printf("Test " #x " failed (in field %d).\n", field->tag); \
+    return false; \
+    }
+
+static bool read_varint(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    uint64_t value;
+    if (!pb_decode_varint(stream, &value))
+        return false;
+    
+    TEST((int64_t)value == (long)*arg);
+    return true;
+}
+
+static bool read_svarint(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    int64_t value;
+    if (!pb_decode_svarint(stream, &value))
+        return false;
+    
+    TEST(value == (long)*arg);
+    return true;
+}
+
+static bool read_fixed32(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    uint32_t value;
+    if (!pb_decode_fixed32(stream, &value))
+        return false;
+    
+    TEST(value == *(uint32_t*)*arg);
+    return true;
+}
+
+static bool read_fixed64(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    uint64_t value;
+    if (!pb_decode_fixed64(stream, &value))
+        return false;
+    
+    TEST(value == *(uint64_t*)*arg);
+    return true;
+}
+
+static bool read_string(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    uint8_t buf[16] = {0};
+    size_t len = stream->bytes_left;
+    
+    if (len > sizeof(buf) - 1 || !pb_read(stream, buf, len))
+        return false;
+    
+    TEST(strcmp((char*)buf, *arg) == 0);
+    return true;
+}
+
+static bool read_submsg(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    SubMessage submsg = {""};
+    
+    if (!pb_decode(stream, SubMessage_fields, &submsg))
+        return false;
+    
+    TEST(memcmp(&submsg, *arg, sizeof(submsg)));
+    return true;
+}
+
+static bool read_emptymsg(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    EmptyMessage emptymsg = {0};
+    return pb_decode(stream, EmptyMessage_fields, &emptymsg);
+}
+
+static bool read_repeated_varint(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    int32_t** expected = (int32_t**)arg;
+    uint64_t value;
+    if (!pb_decode_varint(stream, &value))
+        return false;
+
+    TEST(*(*expected)++ == value);
+    return true;
+}
+
+static bool read_repeated_svarint(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    int32_t** expected = (int32_t**)arg;
+    int64_t value;
+    if (!pb_decode_svarint(stream, &value))
+        return false;
+
+    TEST(*(*expected)++ == value);
+    return true;
+}
+
+static bool read_repeated_fixed32(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    uint32_t** expected = (uint32_t**)arg;
+    uint32_t value;
+    if (!pb_decode_fixed32(stream, &value))
+        return false;
+
+    TEST(*(*expected)++ == value);
+    return true;
+}
+
+static bool read_repeated_fixed64(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    uint64_t** expected = (uint64_t**)arg;
+    uint64_t value;
+    if (!pb_decode_fixed64(stream, &value))
+        return false;
+
+    TEST(*(*expected)++ == value);
+    return true;
+}
+
+static bool read_repeated_string(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    uint8_t*** expected = (uint8_t***)arg;
+    uint8_t buf[16] = {0};
+    size_t len = stream->bytes_left;
+    
+    if (len > sizeof(buf) - 1 || !pb_read(stream, buf, len))
+        return false;
+    
+    TEST(strcmp((char*)*(*expected)++, (char*)buf) == 0);
+    return true;
+}
+
+static bool read_repeated_submsg(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    SubMessage** expected = (SubMessage**)arg;
+    SubMessage decoded = {""};
+    if (!pb_decode(stream, SubMessage_fields, &decoded))
+        return false;
+
+    TEST(memcmp((*expected)++, &decoded, sizeof(decoded)) == 0);
+    return true;
+}
+
+static bool read_limits(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    Limits decoded = {0};
+    if (!pb_decode(stream, Limits_fields, &decoded))
+        return false;
+
+    TEST(decoded.int32_min  == INT32_MIN);
+    TEST(decoded.int32_max  == INT32_MAX);
+    TEST(decoded.uint32_min == 0);
+    TEST(decoded.uint32_max == UINT32_MAX);
+    TEST(decoded.int64_min  == INT64_MIN);
+    TEST(decoded.int64_max  == INT64_MAX);
+    TEST(decoded.uint64_min == 0);
+    TEST(decoded.uint64_max == UINT64_MAX);
+    TEST(decoded.enum_min   == HugeEnum_Negative);
+    TEST(decoded.enum_max   == HugeEnum_Positive);
+    
+    return true;
+}
+
+/* This function is called once from main(), it handles
+   the decoding and checks the fields. */
+bool check_alltypes(pb_istream_t *stream, int mode)
+{
+    /* Values for use from callbacks through pointers. */
+    uint32_t    req_fixed32     = 1008;
+    int32_t     req_sfixed32    = -1009;
+    float       req_float       = 1010.0f;
+    uint64_t    req_fixed64     = 1011;
+    int64_t     req_sfixed64    = -1012;
+    double      req_double      = 1013.0;
+    SubMessage  req_submsg      = {"1016", 1016};
+    
+    int32_t     rep_int32[5]    = {0, 0, 0, 0, -2001};
+    int32_t     rep_int64[5]    = {0, 0, 0, 0, -2002};
+    int32_t     rep_uint32[5]   = {0, 0, 0, 0,  2003};
+    int32_t     rep_uint64[5]   = {0, 0, 0, 0,  2004};
+    int32_t     rep_sint32[5]   = {0, 0, 0, 0, -2005};
+    int32_t     rep_sint64[5]   = {0, 0, 0, 0, -2006};
+    int32_t     rep_bool[5]     = {false, false, false, false, true};
+    uint32_t    rep_fixed32[5]  = {0, 0, 0, 0,  2008};
+    int32_t     rep_sfixed32[5] = {0, 0, 0, 0, -2009};
+    float       rep_float[5]    = {0, 0, 0, 0,  2010.0f};
+    uint64_t    rep_fixed64[5]  = {0, 0, 0, 0,  2011};
+    int64_t     rep_sfixed64[5] = {0, 0, 0, 0, -2012};
+    double      rep_double[5]   = {0, 0, 0, 0,  2013.0};
+    char*       rep_string[5]   = {"", "", "", "", "2014"};
+    char*       rep_bytes[5]    = {"", "", "", "", "2015"};
+    SubMessage  rep_submsg[5]   = {{"", 0, 0, 3},
+                                   {"", 0, 0, 3},
+                                   {"", 0, 0, 3},
+                                   {"", 0, 0, 3},
+                                   {"2016", 2016, true, 2016}};
+    int32_t     rep_enum[5]     = {0, 0, 0, 0, MyEnum_Truth};
+    
+    uint32_t    opt_fixed32     = 3048;
+    int32_t     opt_sfixed32    = 3049;
+    float       opt_float       = 3050.0f;
+    uint64_t    opt_fixed64     = 3051;
+    int64_t     opt_sfixed64    = 3052;
+    double      opt_double      = 3053.0f;
+    SubMessage  opt_submsg      = {"3056", 3056};
+    
+    /* Bind callbacks for required fields */
+    AllTypes alltypes;
+    
+    /* Fill with garbage to better detect initialization errors */
+    memset(&alltypes, 0xAA, sizeof(alltypes));
+    alltypes.extensions = 0;
+    
+    alltypes.req_int32.funcs.decode = &read_varint;
+    alltypes.req_int32.arg = (void*)-1001;
+    
+    alltypes.req_int64.funcs.decode = &read_varint;
+    alltypes.req_int64.arg = (void*)-1002;
+    
+    alltypes.req_uint32.funcs.decode = &read_varint;
+    alltypes.req_uint32.arg = (void*)1003;
+
+    alltypes.req_uint32.funcs.decode = &read_varint;
+    alltypes.req_uint32.arg = (void*)1003;    
+    
+    alltypes.req_uint64.funcs.decode = &read_varint;
+    alltypes.req_uint64.arg = (void*)1004;
+    
+    alltypes.req_sint32.funcs.decode = &read_svarint;
+    alltypes.req_sint32.arg = (void*)-1005;   
+    
+    alltypes.req_sint64.funcs.decode = &read_svarint;
+    alltypes.req_sint64.arg = (void*)-1006;   
+    
+    alltypes.req_bool.funcs.decode = &read_varint;
+    alltypes.req_bool.arg = (void*)true;   
+    
+    alltypes.req_fixed32.funcs.decode = &read_fixed32;
+    alltypes.req_fixed32.arg = &req_fixed32;
+    
+    alltypes.req_sfixed32.funcs.decode = &read_fixed32;
+    alltypes.req_sfixed32.arg = &req_sfixed32;
+    
+    alltypes.req_float.funcs.decode = &read_fixed32;
+    alltypes.req_float.arg = &req_float;
+    
+    alltypes.req_fixed64.funcs.decode = &read_fixed64;
+    alltypes.req_fixed64.arg = &req_fixed64;
+    
+    alltypes.req_sfixed64.funcs.decode = &read_fixed64;
+    alltypes.req_sfixed64.arg = &req_sfixed64;
+    
+    alltypes.req_double.funcs.decode = &read_fixed64;
+    alltypes.req_double.arg = &req_double;
+    
+    alltypes.req_string.funcs.decode = &read_string;
+    alltypes.req_string.arg = "1014";
+    
+    alltypes.req_bytes.funcs.decode = &read_string;
+    alltypes.req_bytes.arg = "1015";
+    
+    alltypes.req_submsg.funcs.decode = &read_submsg;
+    alltypes.req_submsg.arg = &req_submsg;
+    
+    alltypes.req_enum.funcs.decode = &read_varint;
+    alltypes.req_enum.arg = (void*)MyEnum_Truth;
+    
+    alltypes.req_emptymsg.funcs.decode = &read_emptymsg;
+    
+    /* Bind callbacks for repeated fields */
+    alltypes.rep_int32.funcs.decode = &read_repeated_varint;
+    alltypes.rep_int32.arg = rep_int32;
+    
+    alltypes.rep_int64.funcs.decode = &read_repeated_varint;
+    alltypes.rep_int64.arg = rep_int64;
+    
+    alltypes.rep_uint32.funcs.decode = &read_repeated_varint;
+    alltypes.rep_uint32.arg = rep_uint32;
+    
+    alltypes.rep_uint64.funcs.decode = &read_repeated_varint;
+    alltypes.rep_uint64.arg = rep_uint64;
+    
+    alltypes.rep_sint32.funcs.decode = &read_repeated_svarint;
+    alltypes.rep_sint32.arg = rep_sint32;
+    
+    alltypes.rep_sint64.funcs.decode = &read_repeated_svarint;
+    alltypes.rep_sint64.arg = rep_sint64;
+    
+    alltypes.rep_bool.funcs.decode = &read_repeated_varint;
+    alltypes.rep_bool.arg = rep_bool;
+    
+    alltypes.rep_fixed32.funcs.decode = &read_repeated_fixed32;
+    alltypes.rep_fixed32.arg = rep_fixed32;
+    
+    alltypes.rep_sfixed32.funcs.decode = &read_repeated_fixed32;
+    alltypes.rep_sfixed32.arg = rep_sfixed32;
+    
+    alltypes.rep_float.funcs.decode = &read_repeated_fixed32;
+    alltypes.rep_float.arg = rep_float;
+    
+    alltypes.rep_fixed64.funcs.decode = &read_repeated_fixed64;
+    alltypes.rep_fixed64.arg = rep_fixed64;
+    
+    alltypes.rep_sfixed64.funcs.decode = &read_repeated_fixed64;
+    alltypes.rep_sfixed64.arg = rep_sfixed64;
+    
+    alltypes.rep_double.funcs.decode = &read_repeated_fixed64;
+    alltypes.rep_double.arg = rep_double;
+    
+    alltypes.rep_string.funcs.decode = &read_repeated_string;
+    alltypes.rep_string.arg = rep_string;
+    
+    alltypes.rep_bytes.funcs.decode = &read_repeated_string;
+    alltypes.rep_bytes.arg = rep_bytes;
+    
+    alltypes.rep_submsg.funcs.decode = &read_repeated_submsg;
+    alltypes.rep_submsg.arg = rep_submsg;
+    
+    alltypes.rep_enum.funcs.decode = &read_repeated_varint;
+    alltypes.rep_enum.arg = rep_enum;
+    
+    alltypes.rep_emptymsg.funcs.decode = &read_emptymsg;
+    
+    alltypes.req_limits.funcs.decode = &read_limits;
+    
+    alltypes.end.funcs.decode = &read_varint;
+    alltypes.end.arg = (void*)1099;
+    
+    /* Bind callbacks for optional fields */
+    if (mode == 1)
+    {
+        alltypes.opt_int32.funcs.decode = &read_varint;
+        alltypes.opt_int32.arg = (void*)3041;
+        
+        alltypes.opt_int64.funcs.decode = &read_varint;
+        alltypes.opt_int64.arg = (void*)3042;
+        
+        alltypes.opt_uint32.funcs.decode = &read_varint;
+        alltypes.opt_uint32.arg = (void*)3043;
+        
+        alltypes.opt_uint64.funcs.decode = &read_varint;
+        alltypes.opt_uint64.arg = (void*)3044;
+        
+        alltypes.opt_sint32.funcs.decode = &read_svarint;
+        alltypes.opt_sint32.arg = (void*)3045;
+        
+        alltypes.opt_sint64.funcs.decode = &read_svarint;
+        alltypes.opt_sint64.arg = (void*)3046;
+        
+        alltypes.opt_bool.funcs.decode = &read_varint;
+        alltypes.opt_bool.arg = (void*)true;
+
+        alltypes.opt_fixed32.funcs.decode = &read_fixed32;
+        alltypes.opt_fixed32.arg = &opt_fixed32;
+        
+        alltypes.opt_sfixed32.funcs.decode = &read_fixed32;
+        alltypes.opt_sfixed32.arg = &opt_sfixed32;
+        
+        alltypes.opt_float.funcs.decode = &read_fixed32;
+        alltypes.opt_float.arg = &opt_float;
+        
+        alltypes.opt_fixed64.funcs.decode = &read_fixed64;
+        alltypes.opt_fixed64.arg = &opt_fixed64;
+        
+        alltypes.opt_sfixed64.funcs.decode = &read_fixed64;
+        alltypes.opt_sfixed64.arg = &opt_sfixed64;
+        
+        alltypes.opt_double.funcs.decode = &read_fixed64;
+        alltypes.opt_double.arg = &opt_double;
+        
+        alltypes.opt_string.funcs.decode = &read_string;
+        alltypes.opt_string.arg = "3054";
+        
+        alltypes.opt_bytes.funcs.decode = &read_string;
+        alltypes.opt_bytes.arg = "3055";
+        
+        alltypes.opt_submsg.funcs.decode = &read_submsg;
+        alltypes.opt_submsg.arg = &opt_submsg;
+        
+        alltypes.opt_enum.funcs.decode = &read_varint;
+        alltypes.opt_enum.arg = (void*)MyEnum_Truth;
+        
+        alltypes.opt_emptymsg.funcs.decode = &read_emptymsg;
+    }
+    
+    return pb_decode(stream, AllTypes_fields, &alltypes);
+}
+
+int main(int argc, char **argv)
+{
+    uint8_t buffer[1024];
+    size_t count;
+    pb_istream_t stream;
+
+    /* Whether to expect the optional values or the default values. */
+    int mode = (argc > 1) ? atoi(argv[1]) : 0;
+    
+    /* Read the data into buffer */
+    SET_BINARY_MODE(stdin);
+    count = fread(buffer, 1, sizeof(buffer), stdin);
+    
+    /* Construct a pb_istream_t for reading from the buffer */
+    stream = pb_istream_from_buffer(buffer, count);
+    
+    /* Decode and print out the stuff */
+    if (!check_alltypes(&stream, mode))
+    {
+        printf("Parsing failed: %s\n", PB_GET_ERROR(&stream));
+        return 1;
+    } else {
+        return 0;
+    }
+}

tests/alltypes_callback/encode_alltypes_callback.c

@@ -0,0 +1,397 @@
+/* Attempts to test all the datatypes supported by ProtoBuf when used as callback fields.
+ * Note that normally there would be no reason to use callback fields for this,
+ * because each encoder defined here only gives a single field.
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <pb_encode.h>
+#include "alltypes.pb.h"
+#include "test_helpers.h"
+
+static bool write_varint(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_varint(stream, (long)*arg);
+}
+
+static bool write_svarint(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_svarint(stream, (long)*arg);
+}
+
+static bool write_fixed32(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_fixed32(stream, *arg);
+}
+
+static bool write_fixed64(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_fixed64(stream, *arg);
+}
+
+static bool write_string(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_string(stream, *arg, strlen(*arg));
+}
+
+static bool write_submsg(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+   
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, SubMessage_fields, *arg);
+}
+
+static bool write_emptymsg(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    EmptyMessage emptymsg = {0};
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, EmptyMessage_fields, &emptymsg);
+}
+
+static bool write_repeated_varint(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_varint(stream, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_varint(stream, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_varint(stream, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_varint(stream, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_varint(stream, (long)*arg);
+}
+
+static bool write_repeated_svarint(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_svarint(stream, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_svarint(stream, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_svarint(stream, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_svarint(stream, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_svarint(stream, (long)*arg);
+}
+
+static bool write_repeated_fixed32(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    uint32_t dummy = 0;
+
+    /* Make it a packed field */
+    return pb_encode_tag(stream, PB_WT_STRING, field->tag) &&
+           pb_encode_varint(stream, 5 * 4) && /* Number of bytes */
+           pb_encode_fixed32(stream, &dummy) &&
+           pb_encode_fixed32(stream, &dummy) &&
+           pb_encode_fixed32(stream, &dummy) &&
+           pb_encode_fixed32(stream, &dummy) &&
+           pb_encode_fixed32(stream, *arg);
+}
+
+static bool write_repeated_fixed64(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    uint64_t dummy = 0;
+
+    /* Make it a packed field */
+    return pb_encode_tag(stream, PB_WT_STRING, field->tag) &&
+           pb_encode_varint(stream, 5 * 8) && /* Number of bytes */
+           pb_encode_fixed64(stream, &dummy) &&
+           pb_encode_fixed64(stream, &dummy) &&
+           pb_encode_fixed64(stream, &dummy) &&
+           pb_encode_fixed64(stream, &dummy) &&
+           pb_encode_fixed64(stream, *arg);
+}
+
+static bool write_repeated_string(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_string(stream, 0, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_string(stream, 0, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_string(stream, 0, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_string(stream, 0, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_string(stream, *arg, strlen(*arg));
+}
+
+static bool write_repeated_submsg(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    SubMessage dummy = {""};
+
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, SubMessage_fields, &dummy) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, SubMessage_fields, &dummy) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, SubMessage_fields, &dummy) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, SubMessage_fields, &dummy) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, SubMessage_fields, *arg);
+}
+
+static bool write_limits(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    Limits limits = {0};
+    limits.int32_min  = INT32_MIN;
+    limits.int32_max  = INT32_MAX;
+    limits.uint32_min = 0;
+    limits.uint32_max = UINT32_MAX;
+    limits.int64_min  = INT64_MIN;
+    limits.int64_max  = INT64_MAX;
+    limits.uint64_min = 0;
+    limits.uint64_max = UINT64_MAX;
+    limits.enum_min   = HugeEnum_Negative;
+    limits.enum_max   = HugeEnum_Positive;
+   
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, Limits_fields, &limits);
+}
+
+static bool write_repeated_emptymsg(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    EmptyMessage emptymsg = {0};
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, EmptyMessage_fields, &emptymsg) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, EmptyMessage_fields, &emptymsg) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, EmptyMessage_fields, &emptymsg) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, EmptyMessage_fields, &emptymsg) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, EmptyMessage_fields, &emptymsg);
+}
+
+int main(int argc, char **argv)
+{
+    int mode = (argc > 1) ? atoi(argv[1]) : 0;
+
+    /* Values for use from callbacks through pointers. */
+    uint32_t    req_fixed32     = 1008;
+    int32_t     req_sfixed32    = -1009;
+    float       req_float       = 1010.0f;
+    uint64_t    req_fixed64     = 1011;
+    int64_t     req_sfixed64    = -1012;
+    double      req_double      = 1013.0;
+    SubMessage  req_submsg      = {"1016", 1016};
+    
+    uint32_t    rep_fixed32     = 2008;
+    int32_t     rep_sfixed32    = -2009;
+    float       rep_float       = 2010.0f;
+    uint64_t    rep_fixed64     = 2011;
+    int64_t     rep_sfixed64    = -2012;
+    double      rep_double      = 2013.0;
+    SubMessage  rep_submsg      = {"2016", 2016, true, 2016};
+    
+    uint32_t    opt_fixed32     = 3048;
+    int32_t     opt_sfixed32    = 3049;
+    float       opt_float       = 3050.0f;
+    uint64_t    opt_fixed64     = 3051;
+    int64_t     opt_sfixed64    = 3052;
+    double      opt_double      = 3053.0f;
+    SubMessage  opt_submsg      = {"3056", 3056};
+    
+    /* Bind callbacks for required fields */
+    AllTypes alltypes = {{{0}}};
+    
+    alltypes.req_int32.funcs.encode = &write_varint;
+    alltypes.req_int32.arg = (void*)-1001;
+    
+    alltypes.req_int64.funcs.encode = &write_varint;
+    alltypes.req_int64.arg = (void*)-1002;
+    
+    alltypes.req_uint32.funcs.encode = &write_varint;
+    alltypes.req_uint32.arg = (void*)1003;
+
+    alltypes.req_uint32.funcs.encode = &write_varint;
+    alltypes.req_uint32.arg = (void*)1003;    
+    
+    alltypes.req_uint64.funcs.encode = &write_varint;
+    alltypes.req_uint64.arg = (void*)1004;
+    
+    alltypes.req_sint32.funcs.encode = &write_svarint;
+    alltypes.req_sint32.arg = (void*)-1005;   
+    
+    alltypes.req_sint64.funcs.encode = &write_svarint;
+    alltypes.req_sint64.arg = (void*)-1006;   
+    
+    alltypes.req_bool.funcs.encode = &write_varint;
+    alltypes.req_bool.arg = (void*)true;   
+    
+    alltypes.req_fixed32.funcs.encode = &write_fixed32;
+    alltypes.req_fixed32.arg = &req_fixed32;
+    
+    alltypes.req_sfixed32.funcs.encode = &write_fixed32;
+    alltypes.req_sfixed32.arg = &req_sfixed32;
+    
+    alltypes.req_float.funcs.encode = &write_fixed32;
+    alltypes.req_float.arg = &req_float;
+    
+    alltypes.req_fixed64.funcs.encode = &write_fixed64;
+    alltypes.req_fixed64.arg = &req_fixed64;
+    
+    alltypes.req_sfixed64.funcs.encode = &write_fixed64;
+    alltypes.req_sfixed64.arg = &req_sfixed64;
+    
+    alltypes.req_double.funcs.encode = &write_fixed64;
+    alltypes.req_double.arg = &req_double;
+    
+    alltypes.req_string.funcs.encode = &write_string;
+    alltypes.req_string.arg = "1014";
+    
+    alltypes.req_bytes.funcs.encode = &write_string;
+    alltypes.req_bytes.arg = "1015";
+    
+    alltypes.req_submsg.funcs.encode = &write_submsg;
+    alltypes.req_submsg.arg = &req_submsg;
+    
+    alltypes.req_enum.funcs.encode = &write_varint;
+    alltypes.req_enum.arg = (void*)MyEnum_Truth;
+    
+    alltypes.req_emptymsg.funcs.encode = &write_emptymsg;
+    
+    /* Bind callbacks for repeated fields */
+    alltypes.rep_int32.funcs.encode = &write_repeated_varint;
+    alltypes.rep_int32.arg = (void*)-2001;
+    
+    alltypes.rep_int64.funcs.encode = &write_repeated_varint;
+    alltypes.rep_int64.arg = (void*)-2002;
+    
+    alltypes.rep_uint32.funcs.encode = &write_repeated_varint;
+    alltypes.rep_uint32.arg = (void*)2003;
+    
+    alltypes.rep_uint64.funcs.encode = &write_repeated_varint;
+    alltypes.rep_uint64.arg = (void*)2004;
+    
+    alltypes.rep_sint32.funcs.encode = &write_repeated_svarint;
+    alltypes.rep_sint32.arg = (void*)-2005;
+    
+    alltypes.rep_sint64.funcs.encode = &write_repeated_svarint;
+    alltypes.rep_sint64.arg = (void*)-2006;
+    
+    alltypes.rep_bool.funcs.encode = &write_repeated_varint;
+    alltypes.rep_bool.arg = (void*)true;
+    
+    alltypes.rep_fixed32.funcs.encode = &write_repeated_fixed32;
+    alltypes.rep_fixed32.arg = &rep_fixed32;
+
+    alltypes.rep_sfixed32.funcs.encode = &write_repeated_fixed32;
+    alltypes.rep_sfixed32.arg = &rep_sfixed32;
+    
+    alltypes.rep_float.funcs.encode = &write_repeated_fixed32;
+    alltypes.rep_float.arg = &rep_float;
+    
+    alltypes.rep_fixed64.funcs.encode = &write_repeated_fixed64;
+    alltypes.rep_fixed64.arg = &rep_fixed64;
+
+    alltypes.rep_sfixed64.funcs.encode = &write_repeated_fixed64;
+    alltypes.rep_sfixed64.arg = &rep_sfixed64;
+    
+    alltypes.rep_double.funcs.encode = &write_repeated_fixed64;
+    alltypes.rep_double.arg = &rep_double;
+    
+    alltypes.rep_string.funcs.encode = &write_repeated_string;
+    alltypes.rep_string.arg = "2014";
+    
+    alltypes.rep_bytes.funcs.encode = &write_repeated_string;
+    alltypes.rep_bytes.arg = "2015";
+    
+    alltypes.rep_submsg.funcs.encode = &write_repeated_submsg;
+    alltypes.rep_submsg.arg = &rep_submsg;
+    
+    alltypes.rep_enum.funcs.encode = &write_repeated_varint;
+    alltypes.rep_enum.arg = (void*)MyEnum_Truth;
+    
+    alltypes.rep_emptymsg.funcs.encode = &write_repeated_emptymsg;
+    
+    alltypes.req_limits.funcs.encode = &write_limits;
+    
+    /* Bind callbacks for optional fields */
+    if (mode != 0)
+    {
+        alltypes.opt_int32.funcs.encode = &write_varint;
+        alltypes.opt_int32.arg = (void*)3041;
+        
+        alltypes.opt_int64.funcs.encode = &write_varint;
+        alltypes.opt_int64.arg = (void*)3042;
+        
+        alltypes.opt_uint32.funcs.encode = &write_varint;
+        alltypes.opt_uint32.arg = (void*)3043;
+        
+        alltypes.opt_uint64.funcs.encode = &write_varint;
+        alltypes.opt_uint64.arg = (void*)3044;
+        
+        alltypes.opt_sint32.funcs.encode = &write_svarint;
+        alltypes.opt_sint32.arg = (void*)3045;
+        
+        alltypes.opt_sint64.funcs.encode = &write_svarint;
+        alltypes.opt_sint64.arg = (void*)3046;
+        
+        alltypes.opt_bool.funcs.encode = &write_varint;
+        alltypes.opt_bool.arg = (void*)true;
+
+        alltypes.opt_fixed32.funcs.encode = &write_fixed32;
+        alltypes.opt_fixed32.arg = &opt_fixed32;
+        
+        alltypes.opt_sfixed32.funcs.encode = &write_fixed32;
+        alltypes.opt_sfixed32.arg = &opt_sfixed32;
+        
+        alltypes.opt_float.funcs.encode = &write_fixed32;
+        alltypes.opt_float.arg = &opt_float;
+        
+        alltypes.opt_fixed64.funcs.encode = &write_fixed64;
+        alltypes.opt_fixed64.arg = &opt_fixed64;
+        
+        alltypes.opt_sfixed64.funcs.encode = &write_fixed64;
+        alltypes.opt_sfixed64.arg = &opt_sfixed64;
+        
+        alltypes.opt_double.funcs.encode = &write_fixed64;
+        alltypes.opt_double.arg = &opt_double;
+        
+        alltypes.opt_string.funcs.encode = &write_string;
+        alltypes.opt_string.arg = "3054";
+        
+        alltypes.opt_bytes.funcs.encode = &write_string;
+        alltypes.opt_bytes.arg = "3055";
+        
+        alltypes.opt_submsg.funcs.encode = &write_submsg;
+        alltypes.opt_submsg.arg = &opt_submsg;
+        
+        alltypes.opt_enum.funcs.encode = &write_varint;
+        alltypes.opt_enum.arg = (void*)MyEnum_Truth;
+        
+        alltypes.opt_emptymsg.funcs.encode = &write_emptymsg;
+    }
+    
+    alltypes.end.funcs.encode = &write_varint;
+    alltypes.end.arg = (void*)1099;
+    
+    {
+        uint8_t buffer[2048];
+        pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
+        
+        /* Now encode it and check if we succeeded. */
+        if (pb_encode(&stream, AllTypes_fields, &alltypes))
+        {
+            SET_BINARY_MODE(stdout);
+            fwrite(buffer, 1, stream.bytes_written, stdout);
+            return 0; /* Success */
+        }
+        else
+        {
+            fprintf(stderr, "Encoding failed: %s\n", PB_GET_ERROR(&stream));
+            return 1; /* Failure */
+        }
+    }
+}

tests/alltypes_pointer/SConscript

@@ -0,0 +1,49 @@
+# Encode the AllTypes message using pointers for all fields, and verify the
+# output against the normal AllTypes test case.
+
+Import("env")
+
+# We need our own pb_decode.o for the malloc support
+env = env.Clone()
+env.Append(CPPDEFINES = {'PB_ENABLE_MALLOC': 1});
+
+# Disable libmudflap, because it will confuse valgrind
+# and other memory leak detection tools.
+if '-fmudflap' in env["CCFLAGS"]:
+    env["CCFLAGS"].remove("-fmudflap")
+    env["LINKFLAGS"].remove("-fmudflap")
+    env["LIBS"].remove("mudflap")
+
+strict = env.Clone()
+strict.Append(CFLAGS = strict['CORECFLAGS'])
+strict.Object("pb_decode_with_malloc.o", "$NANOPB/pb_decode.c")
+strict.Object("pb_encode_with_malloc.o", "$NANOPB/pb_encode.c")
+strict.Object("pb_common_with_malloc.o", "$NANOPB/pb_common.c")
+
+c = Copy("$TARGET", "$SOURCE")
+env.Command("alltypes.proto", "#alltypes/alltypes.proto", c)
+
+env.NanopbProto(["alltypes", "alltypes.options"])
+enc = env.Program(["encode_alltypes_pointer.c", "alltypes.pb.c", "pb_encode_with_malloc.o", "pb_common_with_malloc.o"])
+dec = env.Program(["decode_alltypes_pointer.c", "alltypes.pb.c", "pb_decode_with_malloc.o", "pb_common_with_malloc.o"])
+
+# Encode and compare results to non-pointer alltypes test case
+env.RunTest(enc)
+env.Compare(["encode_alltypes_pointer.output", "$BUILD/alltypes/encode_alltypes.output"])
+
+# Decode (under valgrind if available)
+valgrind = env.WhereIs('valgrind')
+kwargs = {}
+if valgrind:
+    kwargs['COMMAND'] = valgrind
+    kwargs['ARGS'] = ["-q", dec[0].abspath]
+
+env.RunTest("decode_alltypes.output", [dec, "encode_alltypes_pointer.output"], **kwargs)
+
+# Do the same thing with the optional fields present
+env.RunTest("optionals.output", enc, ARGS = ['1'])
+env.Compare(["optionals.output", "$BUILD/alltypes/optionals.output"])
+
+kwargs['ARGS'] = kwargs.get('ARGS', []) + ['1']
+env.RunTest("optionals.decout", [dec, "optionals.output"], **kwargs)
+

tests/alltypes_pointer/alltypes.options

@@ -0,0 +1,3 @@
+# Generate all fields as pointers.
+* type:FT_POINTER
+

tests/alltypes_pointer/decode_alltypes_pointer.c

@@ -0,0 +1,174 @@
+#include <stdio.h>
+#include <string.h>
+#include <stdlib.h>
+#include <pb_decode.h>
+#include "alltypes.pb.h"
+#include "test_helpers.h"
+
+#define TEST(x) if (!(x)) { \
+    fprintf(stderr, "Test " #x " failed.\n"); \
+    status = false; \
+    }
+
+/* This function is called once from main(), it handles
+   the decoding and checks the fields. */
+bool check_alltypes(pb_istream_t *stream, int mode)
+{
+    bool status = true;
+    AllTypes alltypes;
+    
+    /* Fill with garbage to better detect initialization errors */
+    memset(&alltypes, 0xAA, sizeof(alltypes));
+    alltypes.extensions = 0;
+    
+    if (!pb_decode(stream, AllTypes_fields, &alltypes))
+        return false;
+    
+    TEST(alltypes.req_int32     && *alltypes.req_int32         == -1001);
+    TEST(alltypes.req_int64     && *alltypes.req_int64         == -1002);
+    TEST(alltypes.req_uint32    && *alltypes.req_uint32        == 1003);
+    TEST(alltypes.req_uint64    && *alltypes.req_uint64        == 1004);
+    TEST(alltypes.req_sint32    && *alltypes.req_sint32        == -1005);
+    TEST(alltypes.req_sint64    && *alltypes.req_sint64        == -1006);
+    TEST(alltypes.req_bool      && *alltypes.req_bool          == true);
+    
+    TEST(alltypes.req_fixed32   && *alltypes.req_fixed32       == 1008);
+    TEST(alltypes.req_sfixed32  && *alltypes.req_sfixed32      == -1009);
+    TEST(alltypes.req_float     && *alltypes.req_float         == 1010.0f);
+    
+    TEST(alltypes.req_fixed64   && *alltypes.req_fixed64       == 1011);
+    TEST(alltypes.req_sfixed64  && *alltypes.req_sfixed64      == -1012);
+    TEST(alltypes.req_double    && *alltypes.req_double        == 1013.0f);
+    
+    TEST(alltypes.req_string    && strcmp(alltypes.req_string, "1014") == 0);
+    TEST(alltypes.req_bytes     && alltypes.req_bytes->size == 4);
+    TEST(alltypes.req_bytes     && memcmp(&alltypes.req_bytes->bytes, "1015", 4) == 0);
+    TEST(alltypes.req_submsg    && alltypes.req_submsg->substuff1
+                                && strcmp(alltypes.req_submsg->substuff1, "1016") == 0);
+    TEST(alltypes.req_submsg    && alltypes.req_submsg->substuff2
+                                && *alltypes.req_submsg->substuff2 == 1016);
+    TEST(*alltypes.req_enum == MyEnum_Truth);
+
+    TEST(alltypes.rep_int32_count == 5 && alltypes.rep_int32[4] == -2001 && alltypes.rep_int32[0] == 0);
+    TEST(alltypes.rep_int64_count == 5 && alltypes.rep_int64[4] == -2002 && alltypes.rep_int64[0] == 0);
+    TEST(alltypes.rep_uint32_count == 5 && alltypes.rep_uint32[4] == 2003 && alltypes.rep_uint32[0] == 0);
+    TEST(alltypes.rep_uint64_count == 5 && alltypes.rep_uint64[4] == 2004 && alltypes.rep_uint64[0] == 0);
+    TEST(alltypes.rep_sint32_count == 5 && alltypes.rep_sint32[4] == -2005 && alltypes.rep_sint32[0] == 0);
+    TEST(alltypes.rep_sint64_count == 5 && alltypes.rep_sint64[4] == -2006 && alltypes.rep_sint64[0] == 0);
+    TEST(alltypes.rep_bool_count == 5 && alltypes.rep_bool[4] == true && alltypes.rep_bool[0] == false);
+    
+    TEST(alltypes.rep_fixed32_count == 5 && alltypes.rep_fixed32[4] == 2008 && alltypes.rep_fixed32[0] == 0);
+    TEST(alltypes.rep_sfixed32_count == 5 && alltypes.rep_sfixed32[4] == -2009 && alltypes.rep_sfixed32[0] == 0);
+    TEST(alltypes.rep_float_count == 5 && alltypes.rep_float[4] == 2010.0f && alltypes.rep_float[0] == 0.0f);
+    
+    TEST(alltypes.rep_fixed64_count == 5 && alltypes.rep_fixed64[4] == 2011 && alltypes.rep_fixed64[0] == 0);
+    TEST(alltypes.rep_sfixed64_count == 5 && alltypes.rep_sfixed64[4] == -2012 && alltypes.rep_sfixed64[0] == 0);
+    TEST(alltypes.rep_double_count == 5 && alltypes.rep_double[4] == 2013.0 && alltypes.rep_double[0] == 0.0);
+    
+    TEST(alltypes.rep_string_count == 5 && strcmp(alltypes.rep_string[4], "2014") == 0 && alltypes.rep_string[0][0] == '\0');
+    TEST(alltypes.rep_bytes_count == 5 && alltypes.rep_bytes[4]->size == 4 && alltypes.rep_bytes[0]->size == 0);
+    TEST(memcmp(&alltypes.rep_bytes[4]->bytes, "2015", 4) == 0);
+
+    TEST(alltypes.rep_submsg_count == 5);
+    TEST(strcmp(alltypes.rep_submsg[4].substuff1, "2016") == 0 && alltypes.rep_submsg[0].substuff1[0] == '\0');
+    TEST(*alltypes.rep_submsg[4].substuff2 == 2016 && *alltypes.rep_submsg[0].substuff2 == 0);
+    TEST(*alltypes.rep_submsg[4].substuff3 == 2016 && alltypes.rep_submsg[0].substuff3 == NULL);
+    
+    TEST(alltypes.rep_enum_count == 5 && alltypes.rep_enum[4] == MyEnum_Truth && alltypes.rep_enum[0] == MyEnum_Zero);
+    TEST(alltypes.rep_emptymsg_count == 5);
+
+    if (mode == 0)
+    {
+        /* Expect that optional values are not present */
+        TEST(alltypes.opt_int32         == NULL);
+        TEST(alltypes.opt_int64         == NULL);
+        TEST(alltypes.opt_uint32        == NULL);
+        TEST(alltypes.opt_uint64        == NULL);
+        TEST(alltypes.opt_sint32        == NULL);
+        TEST(alltypes.opt_sint64        == NULL);
+        TEST(alltypes.opt_bool          == NULL);
+        
+        TEST(alltypes.opt_fixed32       == NULL);
+        TEST(alltypes.opt_sfixed32      == NULL);
+        TEST(alltypes.opt_float         == NULL);
+        TEST(alltypes.opt_fixed64       == NULL);
+        TEST(alltypes.opt_sfixed64      == NULL);
+        TEST(alltypes.opt_double        == NULL);
+        
+        TEST(alltypes.opt_string        == NULL);
+        TEST(alltypes.opt_bytes         == NULL);
+        TEST(alltypes.opt_submsg        == NULL);
+        TEST(alltypes.opt_enum          == NULL);
+    }
+    else
+    {
+        /* Expect filled-in values */
+        TEST(alltypes.opt_int32 && *alltypes.opt_int32      == 3041);
+        TEST(alltypes.opt_int64 && *alltypes.opt_int64      == 3042);
+        TEST(alltypes.opt_uint32 && *alltypes.opt_uint32    == 3043);
+        TEST(alltypes.opt_uint64 && *alltypes.opt_uint64    == 3044);
+        TEST(alltypes.opt_sint32 && *alltypes.opt_sint32    == 3045);
+        TEST(alltypes.opt_sint64 && *alltypes.opt_sint64    == 3046);
+        TEST(alltypes.opt_bool && *alltypes.opt_bool        == true);
+        
+        TEST(alltypes.opt_fixed32 && *alltypes.opt_fixed32  == 3048);
+        TEST(alltypes.opt_sfixed32 && *alltypes.opt_sfixed32== 3049);
+        TEST(alltypes.opt_float && *alltypes.opt_float      == 3050.0f);
+        TEST(alltypes.opt_fixed64 && *alltypes.opt_fixed64  == 3051);
+        TEST(alltypes.opt_sfixed64 && *alltypes.opt_sfixed64== 3052);
+        TEST(alltypes.opt_double && *alltypes.opt_double    == 3053.0);
+        
+        TEST(alltypes.opt_string && strcmp(alltypes.opt_string, "3054") == 0);
+        TEST(alltypes.opt_bytes && alltypes.opt_bytes->size == 4);
+        TEST(alltypes.opt_bytes && memcmp(&alltypes.opt_bytes->bytes, "3055", 4) == 0);
+        TEST(alltypes.opt_submsg && strcmp(alltypes.opt_submsg->substuff1, "3056") == 0);
+        TEST(alltypes.opt_submsg && *alltypes.opt_submsg->substuff2 == 3056);
+        TEST(alltypes.opt_enum && *alltypes.opt_enum == MyEnum_Truth);
+        TEST(alltypes.opt_emptymsg);
+    }
+    
+    TEST(alltypes.req_limits->int32_min && *alltypes.req_limits->int32_min   == INT32_MIN);
+    TEST(alltypes.req_limits->int32_max && *alltypes.req_limits->int32_max   == INT32_MAX);
+    TEST(alltypes.req_limits->uint32_min && *alltypes.req_limits->uint32_min == 0);
+    TEST(alltypes.req_limits->uint32_max && *alltypes.req_limits->uint32_max == UINT32_MAX);
+    TEST(alltypes.req_limits->int64_min && *alltypes.req_limits->int64_min   == INT64_MIN);
+    TEST(alltypes.req_limits->int64_max && *alltypes.req_limits->int64_max   == INT64_MAX);
+    TEST(alltypes.req_limits->uint64_min && *alltypes.req_limits->uint64_min == 0);
+    TEST(alltypes.req_limits->uint64_max && *alltypes.req_limits->uint64_max == UINT64_MAX);
+    TEST(alltypes.req_limits->enum_min && *alltypes.req_limits->enum_min     == HugeEnum_Negative);
+    TEST(alltypes.req_limits->enum_max && *alltypes.req_limits->enum_max     == HugeEnum_Positive);
+    
+    TEST(alltypes.end && *alltypes.end == 1099);
+
+    pb_release(AllTypes_fields, &alltypes);
+
+    return status;
+}
+
+int main(int argc, char **argv)
+{
+    uint8_t buffer[1024];
+    size_t count;
+    pb_istream_t stream;
+    
+    /* Whether to expect the optional values or the default values. */
+    int mode = (argc > 1) ? atoi(argv[1]) : 0;
+    
+    /* Read the data into buffer */
+    SET_BINARY_MODE(stdin);
+    count = fread(buffer, 1, sizeof(buffer), stdin);
+    
+    /* Construct a pb_istream_t for reading from the buffer */
+    stream = pb_istream_from_buffer(buffer, count);
+    
+    /* Decode and verify the message */
+    if (!check_alltypes(&stream, mode))
+    {
+        fprintf(stderr, "Test failed: %s\n", PB_GET_ERROR(&stream));
+        return 1;
+    }
+    else
+    {
+        return 0;
+    }
+}

tests/alltypes_pointer/encode_alltypes_pointer.c

@@ -0,0 +1,188 @@
+/* Attempts to test all the datatypes supported by ProtoBuf.
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <pb_encode.h>
+#include "alltypes.pb.h"
+#include "test_helpers.h"
+
+int main(int argc, char **argv)
+{
+    int mode = (argc > 1) ? atoi(argv[1]) : 0;
+    
+    /* Values for required fields */
+    int32_t     req_int32         = -1001;
+    int64_t     req_int64         = -1002;
+    uint32_t    req_uint32        = 1003;
+    uint64_t    req_uint64        = 1004;
+    int32_t     req_sint32        = -1005;
+    int64_t     req_sint64        = -1006;
+    bool        req_bool          = true;
+    uint32_t    req_fixed32       = 1008;
+    int32_t     req_sfixed32      = -1009;
+    float       req_float         = 1010.0f;
+    uint64_t    req_fixed64       = 1011;
+    int64_t     req_sfixed64      = -1012;
+    double      req_double        = 1013.0;
+    char*       req_string        = "1014";
+    PB_BYTES_ARRAY_T(4) req_bytes = {4, {'1', '0', '1', '5'}};
+    static int32_t req_substuff   = 1016;
+    SubMessage  req_submsg        = {"1016", &req_substuff};
+    MyEnum      req_enum          = MyEnum_Truth;
+    EmptyMessage req_emptymsg     = {0};
+    
+    int32_t     end               = 1099;
+
+    /* Values for repeated fields */
+    int32_t     rep_int32[5]      = {0, 0, 0, 0, -2001};
+    int64_t     rep_int64[5]      = {0, 0, 0, 0, -2002};
+    uint32_t    rep_uint32[5]     = {0, 0, 0, 0, 2003};
+    uint64_t    rep_uint64[5]     = {0, 0, 0, 0, 2004};
+    int32_t     rep_sint32[5]     = {0, 0, 0, 0, -2005};
+    int64_t     rep_sint64[5]     = {0, 0, 0, 0, -2006};
+    bool        rep_bool[5]       = {false, false, false, false, true};
+    uint32_t    rep_fixed32[5]    = {0, 0, 0, 0, 2008};
+    int32_t     rep_sfixed32[5]   = {0, 0, 0, 0, -2009};
+    float       rep_float[5]      = {0, 0, 0, 0, 2010.0f};
+    uint64_t    rep_fixed64[5]    = {0, 0, 0, 0, 2011};
+    int64_t     rep_sfixed64[5]   = {0, 0, 0, 0, -2012};
+    double      rep_double[5]     = {0, 0, 0, 0, 2013.0f};
+    char*       rep_string[5]     = {"", "", "", "", "2014"};
+    static PB_BYTES_ARRAY_T(4) rep_bytes_4 = {4, {'2', '0', '1', '5'}};
+    pb_bytes_array_t *rep_bytes[5]= {NULL, NULL, NULL, NULL, (pb_bytes_array_t*)&rep_bytes_4};
+    static int32_t rep_sub2zero   = 0;
+    static int32_t rep_substuff2  = 2016;
+    static uint32_t rep_substuff3 = 2016;
+    SubMessage  rep_submsg[5]     = {{"", &rep_sub2zero},
+                                     {"", &rep_sub2zero},
+                                     {"", &rep_sub2zero},
+                                     {"", &rep_sub2zero},
+                                     {"2016", &rep_substuff2, &rep_substuff3}};
+    MyEnum      rep_enum[5]       = {0, 0, 0, 0, MyEnum_Truth};
+    EmptyMessage rep_emptymsg[5]  = {{0}, {0}, {0}, {0}, {0}};
+
+    /* Values for optional fields */
+    int32_t     opt_int32         = 3041;
+    int64_t     opt_int64         = 3042;
+    uint32_t    opt_uint32        = 3043;
+    uint64_t    opt_uint64        = 3044;
+    int32_t     opt_sint32        = 3045;
+    int64_t     opt_sint64        = 3046;
+    bool        opt_bool          = true;
+    uint32_t    opt_fixed32       = 3048;
+    int32_t     opt_sfixed32      = 3049;
+    float       opt_float         = 3050.0f;
+    uint64_t    opt_fixed64       = 3051;
+    int64_t     opt_sfixed64      = 3052;
+    double      opt_double        = 3053.0;
+    char*       opt_string        = "3054";
+    PB_BYTES_ARRAY_T(4) opt_bytes = {4, {'3', '0', '5', '5'}};
+    static int32_t opt_substuff   = 3056;
+    SubMessage  opt_submsg        = {"3056", &opt_substuff};
+    MyEnum      opt_enum          = MyEnum_Truth;
+    EmptyMessage opt_emptymsg     = {0};
+
+    /* Values for the Limits message. */
+    static int32_t  int32_min  = INT32_MIN;
+    static int32_t  int32_max  = INT32_MAX;
+    static uint32_t uint32_min = 0;
+    static uint32_t uint32_max = UINT32_MAX;
+    static int64_t  int64_min  = INT64_MIN;
+    static int64_t  int64_max  = INT64_MAX;
+    static uint64_t uint64_min = 0;
+    static uint64_t uint64_max = UINT64_MAX;
+    static HugeEnum enum_min   = HugeEnum_Negative;
+    static HugeEnum enum_max   = HugeEnum_Positive;
+    Limits req_limits = {&int32_min,    &int32_max,
+                         &uint32_min,   &uint32_max,
+                         &int64_min,    &int64_max,
+                         &uint64_min,   &uint64_max,
+                         &enum_min,     &enum_max};
+
+    /* Initialize the message struct with pointers to the fields. */
+    AllTypes alltypes = {0};
+
+    alltypes.req_int32         = &req_int32;
+    alltypes.req_int64         = &req_int64;
+    alltypes.req_uint32        = &req_uint32;
+    alltypes.req_uint64        = &req_uint64;
+    alltypes.req_sint32        = &req_sint32;
+    alltypes.req_sint64        = &req_sint64;
+    alltypes.req_bool          = &req_bool;
+    alltypes.req_fixed32       = &req_fixed32;
+    alltypes.req_sfixed32      = &req_sfixed32;
+    alltypes.req_float         = &req_float;
+    alltypes.req_fixed64       = &req_fixed64;
+    alltypes.req_sfixed64      = &req_sfixed64;
+    alltypes.req_double        = &req_double;
+    alltypes.req_string        = req_string;
+    alltypes.req_bytes         = (pb_bytes_array_t*)&req_bytes;
+    alltypes.req_submsg        = &req_submsg;
+    alltypes.req_enum          = &req_enum;
+    alltypes.req_emptymsg      = &req_emptymsg;
+    alltypes.req_limits        = &req_limits;
+    
+    alltypes.rep_int32_count    = 5; alltypes.rep_int32     = rep_int32;
+    alltypes.rep_int64_count    = 5; alltypes.rep_int64     = rep_int64;
+    alltypes.rep_uint32_count   = 5; alltypes.rep_uint32    = rep_uint32;
+    alltypes.rep_uint64_count   = 5; alltypes.rep_uint64    = rep_uint64;
+    alltypes.rep_sint32_count   = 5; alltypes.rep_sint32    = rep_sint32;
+    alltypes.rep_sint64_count   = 5; alltypes.rep_sint64    = rep_sint64;
+    alltypes.rep_bool_count     = 5; alltypes.rep_bool      = rep_bool;
+    alltypes.rep_fixed32_count  = 5; alltypes.rep_fixed32   = rep_fixed32;
+    alltypes.rep_sfixed32_count = 5; alltypes.rep_sfixed32  = rep_sfixed32;
+    alltypes.rep_float_count    = 5; alltypes.rep_float     = rep_float;
+    alltypes.rep_fixed64_count  = 5; alltypes.rep_fixed64   = rep_fixed64;
+    alltypes.rep_sfixed64_count = 5; alltypes.rep_sfixed64  = rep_sfixed64;
+    alltypes.rep_double_count   = 5; alltypes.rep_double    = rep_double;
+    alltypes.rep_string_count   = 5; alltypes.rep_string    = rep_string;
+    alltypes.rep_bytes_count    = 5; alltypes.rep_bytes     = rep_bytes;
+    alltypes.rep_submsg_count   = 5; alltypes.rep_submsg    = rep_submsg;
+    alltypes.rep_enum_count     = 5; alltypes.rep_enum      = rep_enum;
+    alltypes.rep_emptymsg_count = 5; alltypes.rep_emptymsg  = rep_emptymsg;
+    
+    if (mode != 0)
+    {
+        /* Fill in values for optional fields */
+        alltypes.opt_int32         = &opt_int32;
+        alltypes.opt_int64         = &opt_int64;
+        alltypes.opt_uint32        = &opt_uint32;
+        alltypes.opt_uint64        = &opt_uint64;
+        alltypes.opt_sint32        = &opt_sint32;
+        alltypes.opt_sint64        = &opt_sint64;
+        alltypes.opt_bool          = &opt_bool;
+        alltypes.opt_fixed32       = &opt_fixed32;
+        alltypes.opt_sfixed32      = &opt_sfixed32;
+        alltypes.opt_float         = &opt_float;
+        alltypes.opt_fixed64       = &opt_fixed64;
+        alltypes.opt_sfixed64      = &opt_sfixed64;
+        alltypes.opt_double        = &opt_double;
+        alltypes.opt_string        = opt_string;
+        alltypes.opt_bytes         = (pb_bytes_array_t*)&opt_bytes;
+        alltypes.opt_submsg        = &opt_submsg;
+        alltypes.opt_enum          = &opt_enum;
+        alltypes.opt_emptymsg      = &opt_emptymsg;
+    }
+    
+    alltypes.end = &end;
+    
+    {
+        uint8_t buffer[4096];
+        pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
+        
+        /* Now encode it and check if we succeeded. */
+        if (pb_encode(&stream, AllTypes_fields, &alltypes))
+        {
+            SET_BINARY_MODE(stdout);
+            fwrite(buffer, 1, stream.bytes_written, stdout);
+            return 0; /* Success */
+        }
+        else
+        {
+            fprintf(stderr, "Encoding failed: %s\n", PB_GET_ERROR(&stream));
+            return 1; /* Failure */
+        }
+    }
+}

tests/backwards_compatibility/SConscript

@@ -0,0 +1,11 @@
+# Check that the old generated .pb.c/.pb.h files are still compatible with the
+# current version of nanopb.
+
+Import("env")
+
+enc = env.Program(["encode_legacy.c", "alltypes_legacy.c", "$COMMON/pb_encode.o", "$COMMON/pb_common.o"])
+dec = env.Program(["decode_legacy.c", "alltypes_legacy.c", "$COMMON/pb_decode.o", "$COMMON/pb_common.o"])
+
+env.RunTest(enc)
+env.RunTest([dec, "encode_legacy.output"])
+

tests/backwards_compatibility/alltypes_legacy.c

@@ -0,0 +1,153 @@
+/* Automatically generated nanopb constant definitions */
+/* Generated by nanopb-0.3.0-dev at Tue Aug 19 17:53:24 2014. */
+
+#include "alltypes_legacy.h"
+
+#if PB_PROTO_HEADER_VERSION != 30
+#error Regenerate this file with the current version of nanopb generator.
+#endif
+
+const char SubMessage_substuff1_default[16] = "1";
+const int32_t SubMessage_substuff2_default = 2;
+const uint32_t SubMessage_substuff3_default = 3u;
+const int32_t Limits_int32_min_default = 2147483647;
+const int32_t Limits_int32_max_default = -2147483647;
+const uint32_t Limits_uint32_min_default = 4294967295u;
+const uint32_t Limits_uint32_max_default = 0u;
+const int64_t Limits_int64_min_default = 9223372036854775807ll;
+const int64_t Limits_int64_max_default = -9223372036854775807ll;
+const uint64_t Limits_uint64_min_default = 18446744073709551615ull;
+const uint64_t Limits_uint64_max_default = 0ull;
+const HugeEnum Limits_enum_min_default = HugeEnum_Positive;
+const HugeEnum Limits_enum_max_default = HugeEnum_Negative;
+const int32_t AllTypes_opt_int32_default = 4041;
+const int64_t AllTypes_opt_int64_default = 4042ll;
+const uint32_t AllTypes_opt_uint32_default = 4043u;
+const uint64_t AllTypes_opt_uint64_default = 4044ull;
+const int32_t AllTypes_opt_sint32_default = 4045;
+const int64_t AllTypes_opt_sint64_default = 4046;
+const bool AllTypes_opt_bool_default = false;
+const uint32_t AllTypes_opt_fixed32_default = 4048u;
+const int32_t AllTypes_opt_sfixed32_default = 4049;
+const float AllTypes_opt_float_default = 4050;
+const uint64_t AllTypes_opt_fixed64_default = 4051ull;
+const int64_t AllTypes_opt_sfixed64_default = 4052ll;
+const double AllTypes_opt_double_default = 4053;
+const char AllTypes_opt_string_default[16] = "4054";
+const AllTypes_opt_bytes_t AllTypes_opt_bytes_default = {4, {0x34,0x30,0x35,0x35}};
+const MyEnum AllTypes_opt_enum_default = MyEnum_Second;
+
+
+const pb_field_t SubMessage_fields[4] = {
+    PB_FIELD(  1, STRING  , REQUIRED, STATIC  , FIRST, SubMessage, substuff1, substuff1, &SubMessage_substuff1_default),
+    PB_FIELD(  2, INT32   , REQUIRED, STATIC  , OTHER, SubMessage, substuff2, substuff1, &SubMessage_substuff2_default),
+    PB_FIELD(  3, FIXED32 , OPTIONAL, STATIC  , OTHER, SubMessage, substuff3, substuff2, &SubMessage_substuff3_default),
+    PB_LAST_FIELD
+};
+
+const pb_field_t EmptyMessage_fields[1] = {
+    PB_LAST_FIELD
+};
+
+const pb_field_t Limits_fields[11] = {
+    PB_FIELD(  1, INT32   , REQUIRED, STATIC  , FIRST, Limits, int32_min, int32_min, &Limits_int32_min_default),
+    PB_FIELD(  2, INT32   , REQUIRED, STATIC  , OTHER, Limits, int32_max, int32_min, &Limits_int32_max_default),
+    PB_FIELD(  3, UINT32  , REQUIRED, STATIC  , OTHER, Limits, uint32_min, int32_max, &Limits_uint32_min_default),
+    PB_FIELD(  4, UINT32  , REQUIRED, STATIC  , OTHER, Limits, uint32_max, uint32_min, &Limits_uint32_max_default),
+    PB_FIELD(  5, INT64   , REQUIRED, STATIC  , OTHER, Limits, int64_min, uint32_max, &Limits_int64_min_default),
+    PB_FIELD(  6, INT64   , REQUIRED, STATIC  , OTHER, Limits, int64_max, int64_min, &Limits_int64_max_default),
+    PB_FIELD(  7, UINT64  , REQUIRED, STATIC  , OTHER, Limits, uint64_min, int64_max, &Limits_uint64_min_default),
+    PB_FIELD(  8, UINT64  , REQUIRED, STATIC  , OTHER, Limits, uint64_max, uint64_min, &Limits_uint64_max_default),
+    PB_FIELD(  9, ENUM    , REQUIRED, STATIC  , OTHER, Limits, enum_min, uint64_max, &Limits_enum_min_default),
+    PB_FIELD( 10, ENUM    , REQUIRED, STATIC  , OTHER, Limits, enum_max, enum_min, &Limits_enum_max_default),
+    PB_LAST_FIELD
+};
+
+const pb_field_t AllTypes_fields[54] = {
+    PB_FIELD(  1, INT32   , REQUIRED, STATIC  , FIRST, AllTypes, req_int32, req_int32, 0),
+    PB_FIELD(  2, INT64   , REQUIRED, STATIC  , OTHER, AllTypes, req_int64, req_int32, 0),
+    PB_FIELD(  3, UINT32  , REQUIRED, STATIC  , OTHER, AllTypes, req_uint32, req_int64, 0),
+    PB_FIELD(  4, UINT64  , REQUIRED, STATIC  , OTHER, AllTypes, req_uint64, req_uint32, 0),
+    PB_FIELD(  5, SINT32  , REQUIRED, STATIC  , OTHER, AllTypes, req_sint32, req_uint64, 0),
+    PB_FIELD(  6, SINT64  , REQUIRED, STATIC  , OTHER, AllTypes, req_sint64, req_sint32, 0),
+    PB_FIELD(  7, BOOL    , REQUIRED, STATIC  , OTHER, AllTypes, req_bool, req_sint64, 0),
+    PB_FIELD(  8, FIXED32 , REQUIRED, STATIC  , OTHER, AllTypes, req_fixed32, req_bool, 0),
+    PB_FIELD(  9, SFIXED32, REQUIRED, STATIC  , OTHER, AllTypes, req_sfixed32, req_fixed32, 0),
+    PB_FIELD( 10, FLOAT   , REQUIRED, STATIC  , OTHER, AllTypes, req_float, req_sfixed32, 0),
+    PB_FIELD( 11, FIXED64 , REQUIRED, STATIC  , OTHER, AllTypes, req_fixed64, req_float, 0),
+    PB_FIELD( 12, SFIXED64, REQUIRED, STATIC  , OTHER, AllTypes, req_sfixed64, req_fixed64, 0),
+    PB_FIELD( 13, DOUBLE  , REQUIRED, STATIC  , OTHER, AllTypes, req_double, req_sfixed64, 0),
+    PB_FIELD( 14, STRING  , REQUIRED, STATIC  , OTHER, AllTypes, req_string, req_double, 0),
+    PB_FIELD( 15, BYTES   , REQUIRED, STATIC  , OTHER, AllTypes, req_bytes, req_string, 0),
+    PB_FIELD( 16, MESSAGE , REQUIRED, STATIC  , OTHER, AllTypes, req_submsg, req_bytes, &SubMessage_fields),
+    PB_FIELD( 17, ENUM    , REQUIRED, STATIC  , OTHER, AllTypes, req_enum, req_submsg, 0),
+    PB_FIELD( 21, INT32   , REPEATED, STATIC  , OTHER, AllTypes, rep_int32, req_enum, 0),
+    PB_FIELD( 22, INT64   , REPEATED, STATIC  , OTHER, AllTypes, rep_int64, rep_int32, 0),
+    PB_FIELD( 23, UINT32  , REPEATED, STATIC  , OTHER, AllTypes, rep_uint32, rep_int64, 0),
+    PB_FIELD( 24, UINT64  , REPEATED, STATIC  , OTHER, AllTypes, rep_uint64, rep_uint32, 0),
+    PB_FIELD( 25, SINT32  , REPEATED, STATIC  , OTHER, AllTypes, rep_sint32, rep_uint64, 0),
+    PB_FIELD( 26, SINT64  , REPEATED, STATIC  , OTHER, AllTypes, rep_sint64, rep_sint32, 0),
+    PB_FIELD( 27, BOOL    , REPEATED, STATIC  , OTHER, AllTypes, rep_bool, rep_sint64, 0),
+    PB_FIELD( 28, FIXED32 , REPEATED, STATIC  , OTHER, AllTypes, rep_fixed32, rep_bool, 0),
+    PB_FIELD( 29, SFIXED32, REPEATED, STATIC  , OTHER, AllTypes, rep_sfixed32, rep_fixed32, 0),
+    PB_FIELD( 30, FLOAT   , REPEATED, STATIC  , OTHER, AllTypes, rep_float, rep_sfixed32, 0),
+    PB_FIELD( 31, FIXED64 , REPEATED, STATIC  , OTHER, AllTypes, rep_fixed64, rep_float, 0),
+    PB_FIELD( 32, SFIXED64, REPEATED, STATIC  , OTHER, AllTypes, rep_sfixed64, rep_fixed64, 0),
+    PB_FIELD( 33, DOUBLE  , REPEATED, STATIC  , OTHER, AllTypes, rep_double, rep_sfixed64, 0),
+    PB_FIELD( 34, STRING  , REPEATED, STATIC  , OTHER, AllTypes, rep_string, rep_double, 0),
+    PB_FIELD( 35, BYTES   , REPEATED, STATIC  , OTHER, AllTypes, rep_bytes, rep_string, 0),
+    PB_FIELD( 36, MESSAGE , REPEATED, STATIC  , OTHER, AllTypes, rep_submsg, rep_bytes, &SubMessage_fields),
+    PB_FIELD( 37, ENUM    , REPEATED, STATIC  , OTHER, AllTypes, rep_enum, rep_submsg, 0),
+    PB_FIELD( 41, INT32   , OPTIONAL, STATIC  , OTHER, AllTypes, opt_int32, rep_enum, &AllTypes_opt_int32_default),
+    PB_FIELD( 42, INT64   , OPTIONAL, STATIC  , OTHER, AllTypes, opt_int64, opt_int32, &AllTypes_opt_int64_default),
+    PB_FIELD( 43, UINT32  , OPTIONAL, STATIC  , OTHER, AllTypes, opt_uint32, opt_int64, &AllTypes_opt_uint32_default),
+    PB_FIELD( 44, UINT64  , OPTIONAL, STATIC  , OTHER, AllTypes, opt_uint64, opt_uint32, &AllTypes_opt_uint64_default),
+    PB_FIELD( 45, SINT32  , OPTIONAL, STATIC  , OTHER, AllTypes, opt_sint32, opt_uint64, &AllTypes_opt_sint32_default),
+    PB_FIELD( 46, SINT64  , OPTIONAL, STATIC  , OTHER, AllTypes, opt_sint64, opt_sint32, &AllTypes_opt_sint64_default),
+    PB_FIELD( 47, BOOL    , OPTIONAL, STATIC  , OTHER, AllTypes, opt_bool, opt_sint64, &AllTypes_opt_bool_default),
+    PB_FIELD( 48, FIXED32 , OPTIONAL, STATIC  , OTHER, AllTypes, opt_fixed32, opt_bool, &AllTypes_opt_fixed32_default),
+    PB_FIELD( 49, SFIXED32, OPTIONAL, STATIC  , OTHER, AllTypes, opt_sfixed32, opt_fixed32, &AllTypes_opt_sfixed32_default),
+    PB_FIELD( 50, FLOAT   , OPTIONAL, STATIC  , OTHER, AllTypes, opt_float, opt_sfixed32, &AllTypes_opt_float_default),
+    PB_FIELD( 51, FIXED64 , OPTIONAL, STATIC  , OTHER, AllTypes, opt_fixed64, opt_float, &AllTypes_opt_fixed64_default),
+    PB_FIELD( 52, SFIXED64, OPTIONAL, STATIC  , OTHER, AllTypes, opt_sfixed64, opt_fixed64, &AllTypes_opt_sfixed64_default),
+    PB_FIELD( 53, DOUBLE  , OPTIONAL, STATIC  , OTHER, AllTypes, opt_double, opt_sfixed64, &AllTypes_opt_double_default),
+    PB_FIELD( 54, STRING  , OPTIONAL, STATIC  , OTHER, AllTypes, opt_string, opt_double, &AllTypes_opt_string_default),
+    PB_FIELD( 55, BYTES   , OPTIONAL, STATIC  , OTHER, AllTypes, opt_bytes, opt_string, &AllTypes_opt_bytes_default),
+    PB_FIELD( 56, MESSAGE , OPTIONAL, STATIC  , OTHER, AllTypes, opt_submsg, opt_bytes, &SubMessage_fields),
+    PB_FIELD( 57, ENUM    , OPTIONAL, STATIC  , OTHER, AllTypes, opt_enum, opt_submsg, &AllTypes_opt_enum_default),
+    PB_FIELD( 99, INT32   , REQUIRED, STATIC  , OTHER, AllTypes, end, opt_enum, 0),
+    PB_FIELD(200, EXTENSION, OPTIONAL, CALLBACK, OTHER, AllTypes, extensions, end, 0),
+    PB_LAST_FIELD
+};
+
+
+/* Check that field information fits in pb_field_t */
+#if !defined(PB_FIELD_32BIT)
+/* If you get an error here, it means that you need to define PB_FIELD_32BIT
+ * compile-time option. You can do that in pb.h or on compiler command line.
+ * 
+ * The reason you need to do this is that some of your messages contain tag
+ * numbers or field sizes that are larger than what can fit in 8 or 16 bit
+ * field descriptors.
+ */
+PB_STATIC_ASSERT((pb_membersize(AllTypes, req_submsg) < 65536 && pb_membersize(AllTypes, rep_submsg[0]) < 65536 && pb_membersize(AllTypes, opt_submsg) < 65536), YOU_MUST_DEFINE_PB_FIELD_32BIT_FOR_MESSAGES_SubMessage_EmptyMessage_Limits_AllTypes)
+#endif
+
+#if !defined(PB_FIELD_16BIT) && !defined(PB_FIELD_32BIT)
+/* If you get an error here, it means that you need to define PB_FIELD_16BIT
+ * compile-time option. You can do that in pb.h or on compiler command line.
+ * 
+ * The reason you need to do this is that some of your messages contain tag
+ * numbers or field sizes that are larger than what can fit in the default
+ * 8 bit descriptors.
+ */
+PB_STATIC_ASSERT((pb_membersize(AllTypes, req_submsg) < 256 && pb_membersize(AllTypes, rep_submsg[0]) < 256 && pb_membersize(AllTypes, opt_submsg) < 256), YOU_MUST_DEFINE_PB_FIELD_16BIT_FOR_MESSAGES_SubMessage_EmptyMessage_Limits_AllTypes)
+#endif
+
+
+/* On some platforms (such as AVR), double is really float.
+ * These are not directly supported by nanopb, but see example_avr_double.
+ * To get rid of this error, remove any double fields from your .proto.
+ */
+PB_STATIC_ASSERT(sizeof(double) == 8, DOUBLE_MUST_BE_8_BYTES)
+

tests/backwards_compatibility/alltypes_legacy.h

@@ -0,0 +1,274 @@
+/* Automatically generated nanopb header */
+/* Generated by nanopb-0.3.0-dev at Tue Aug 19 17:53:24 2014. */
+
+#ifndef PB_ALLTYPES_LEGACY_H_INCLUDED
+#define PB_ALLTYPES_LEGACY_H_INCLUDED
+#include <pb.h>
+
+#if PB_PROTO_HEADER_VERSION != 30
+#error Regenerate this file with the current version of nanopb generator.
+#endif
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/* Enum definitions */
+typedef enum _HugeEnum {
+    HugeEnum_Negative = -2147483647,
+    HugeEnum_Positive = 2147483647
+} HugeEnum;
+
+typedef enum _MyEnum {
+    MyEnum_Zero = 0,
+    MyEnum_First = 1,
+    MyEnum_Second = 2,
+    MyEnum_Truth = 42
+} MyEnum;
+
+/* Struct definitions */
+typedef struct _EmptyMessage {
+    uint8_t dummy_field;
+} EmptyMessage;
+
+typedef struct _Limits {
+    int32_t int32_min;
+    int32_t int32_max;
+    uint32_t uint32_min;
+    uint32_t uint32_max;
+    int64_t int64_min;
+    int64_t int64_max;
+    uint64_t uint64_min;
+    uint64_t uint64_max;
+    HugeEnum enum_min;
+    HugeEnum enum_max;
+} Limits;
+
+typedef struct _SubMessage {
+    char substuff1[16];
+    int32_t substuff2;
+    bool has_substuff3;
+    uint32_t substuff3;
+} SubMessage;
+
+typedef PB_BYTES_ARRAY_T(16) AllTypes_req_bytes_t;
+
+typedef PB_BYTES_ARRAY_T(16) AllTypes_rep_bytes_t;
+
+typedef PB_BYTES_ARRAY_T(16) AllTypes_opt_bytes_t;
+
+typedef struct _AllTypes {
+    int32_t req_int32;
+    int64_t req_int64;
+    uint32_t req_uint32;
+    uint64_t req_uint64;
+    int32_t req_sint32;
+    int64_t req_sint64;
+    bool req_bool;
+    uint32_t req_fixed32;
+    int32_t req_sfixed32;
+    float req_float;
+    uint64_t req_fixed64;
+    int64_t req_sfixed64;
+    double req_double;
+    char req_string[16];
+    AllTypes_req_bytes_t req_bytes;
+    SubMessage req_submsg;
+    MyEnum req_enum;
+    pb_size_t rep_int32_count;
+    int32_t rep_int32[5];
+    pb_size_t rep_int64_count;
+    int64_t rep_int64[5];
+    pb_size_t rep_uint32_count;
+    uint32_t rep_uint32[5];
+    pb_size_t rep_uint64_count;
+    uint64_t rep_uint64[5];
+    pb_size_t rep_sint32_count;
+    int32_t rep_sint32[5];
+    pb_size_t rep_sint64_count;
+    int64_t rep_sint64[5];
+    pb_size_t rep_bool_count;
+    bool rep_bool[5];
+    pb_size_t rep_fixed32_count;
+    uint32_t rep_fixed32[5];
+    pb_size_t rep_sfixed32_count;
+    int32_t rep_sfixed32[5];
+    pb_size_t rep_float_count;
+    float rep_float[5];
+    pb_size_t rep_fixed64_count;
+    uint64_t rep_fixed64[5];
+    pb_size_t rep_sfixed64_count;
+    int64_t rep_sfixed64[5];
+    pb_size_t rep_double_count;
+    double rep_double[5];
+    pb_size_t rep_string_count;
+    char rep_string[5][16];
+    pb_size_t rep_bytes_count;
+    AllTypes_rep_bytes_t rep_bytes[5];
+    pb_size_t rep_submsg_count;
+    SubMessage rep_submsg[5];
+    pb_size_t rep_enum_count;
+    MyEnum rep_enum[5];
+    bool has_opt_int32;
+    int32_t opt_int32;
+    bool has_opt_int64;
+    int64_t opt_int64;
+    bool has_opt_uint32;
+    uint32_t opt_uint32;
+    bool has_opt_uint64;
+    uint64_t opt_uint64;
+    bool has_opt_sint32;
+    int32_t opt_sint32;
+    bool has_opt_sint64;
+    int64_t opt_sint64;
+    bool has_opt_bool;
+    bool opt_bool;
+    bool has_opt_fixed32;
+    uint32_t opt_fixed32;
+    bool has_opt_sfixed32;
+    int32_t opt_sfixed32;
+    bool has_opt_float;
+    float opt_float;
+    bool has_opt_fixed64;
+    uint64_t opt_fixed64;
+    bool has_opt_sfixed64;
+    int64_t opt_sfixed64;
+    bool has_opt_double;
+    double opt_double;
+    bool has_opt_string;
+    char opt_string[16];
+    bool has_opt_bytes;
+    AllTypes_opt_bytes_t opt_bytes;
+    bool has_opt_submsg;
+    SubMessage opt_submsg;
+    bool has_opt_enum;
+    MyEnum opt_enum;
+    int32_t end;
+    pb_extension_t *extensions;
+} AllTypes;
+
+/* Default values for struct fields */
+extern const char SubMessage_substuff1_default[16];
+extern const int32_t SubMessage_substuff2_default;
+extern const uint32_t SubMessage_substuff3_default;
+extern const int32_t Limits_int32_min_default;
+extern const int32_t Limits_int32_max_default;
+extern const uint32_t Limits_uint32_min_default;
+extern const uint32_t Limits_uint32_max_default;
+extern const int64_t Limits_int64_min_default;
+extern const int64_t Limits_int64_max_default;
+extern const uint64_t Limits_uint64_min_default;
+extern const uint64_t Limits_uint64_max_default;
+extern const HugeEnum Limits_enum_min_default;
+extern const HugeEnum Limits_enum_max_default;
+extern const int32_t AllTypes_opt_int32_default;
+extern const int64_t AllTypes_opt_int64_default;
+extern const uint32_t AllTypes_opt_uint32_default;
+extern const uint64_t AllTypes_opt_uint64_default;
+extern const int32_t AllTypes_opt_sint32_default;
+extern const int64_t AllTypes_opt_sint64_default;
+extern const bool AllTypes_opt_bool_default;
+extern const uint32_t AllTypes_opt_fixed32_default;
+extern const int32_t AllTypes_opt_sfixed32_default;
+extern const float AllTypes_opt_float_default;
+extern const uint64_t AllTypes_opt_fixed64_default;
+extern const int64_t AllTypes_opt_sfixed64_default;
+extern const double AllTypes_opt_double_default;
+extern const char AllTypes_opt_string_default[16];
+extern const AllTypes_opt_bytes_t AllTypes_opt_bytes_default;
+extern const MyEnum AllTypes_opt_enum_default;
+
+/* Initializer values for message structs */
+#define SubMessage_init_default                  {"1", 2, false, 3u}
+#define EmptyMessage_init_default                {0}
+#define Limits_init_default                      {2147483647, -2147483647, 4294967295u, 0u, 9223372036854775807ll, -9223372036854775807ll, 18446744073709551615ull, 0ull, HugeEnum_Positive, HugeEnum_Negative}
+#define AllTypes_init_default                    {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, "", {0, {0}}, SubMessage_init_default, (MyEnum)0, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {"", "", "", "", ""}, 0, {{0, {0}}, {0, {0}}, {0, {0}}, {0, {0}}, {0, {0}}}, 0, {SubMessage_init_default, SubMessage_init_default, SubMessage_init_default, SubMessage_init_default, SubMessage_init_default}, 0, {(MyEnum)0, (MyEnum)0, (MyEnum)0, (MyEnum)0, (MyEnum)0}, false, 4041, false, 4042ll, false, 4043u, false, 4044ull, false, 4045, false, 4046, false, false, false, 4048u, false, 4049, false, 4050, false, 4051ull, false, 4052ll, false, 4053, false, "4054", false, {4, {0x34,0x30,0x35,0x35}}, false, SubMessage_init_default, false, MyEnum_Second, 0, NULL}
+#define SubMessage_init_zero                     {"", 0, false, 0}
+#define EmptyMessage_init_zero                   {0}
+#define Limits_init_zero                         {0, 0, 0, 0, 0, 0, 0, 0, (HugeEnum)0, (HugeEnum)0}
+#define AllTypes_init_zero                       {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, "", {0, {0}}, SubMessage_init_zero, (MyEnum)0, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {0, 0, 0, 0, 0}, 0, {"", "", "", "", ""}, 0, {{0, {0}}, {0, {0}}, {0, {0}}, {0, {0}}, {0, {0}}}, 0, {SubMessage_init_zero, SubMessage_init_zero, SubMessage_init_zero, SubMessage_init_zero, SubMessage_init_zero}, 0, {(MyEnum)0, (MyEnum)0, (MyEnum)0, (MyEnum)0, (MyEnum)0}, false, 0, false, 0, false, 0, false, 0, false, 0, false, 0, false, 0, false, 0, false, 0, false, 0, false, 0, false, 0, false, 0, false, "", false, {0, {0}}, false, SubMessage_init_zero, false, (MyEnum)0, 0, NULL}
+
+/* Field tags (for use in manual encoding/decoding) */
+#define Limits_int32_min_tag                     1
+#define Limits_int32_max_tag                     2
+#define Limits_uint32_min_tag                    3
+#define Limits_uint32_max_tag                    4
+#define Limits_int64_min_tag                     5
+#define Limits_int64_max_tag                     6
+#define Limits_uint64_min_tag                    7
+#define Limits_uint64_max_tag                    8
+#define Limits_enum_min_tag                      9
+#define Limits_enum_max_tag                      10
+#define SubMessage_substuff1_tag                 1
+#define SubMessage_substuff2_tag                 2
+#define SubMessage_substuff3_tag                 3
+#define AllTypes_req_int32_tag                   1
+#define AllTypes_req_int64_tag                   2
+#define AllTypes_req_uint32_tag                  3
+#define AllTypes_req_uint64_tag                  4
+#define AllTypes_req_sint32_tag                  5
+#define AllTypes_req_sint64_tag                  6
+#define AllTypes_req_bool_tag                    7
+#define AllTypes_req_fixed32_tag                 8
+#define AllTypes_req_sfixed32_tag                9
+#define AllTypes_req_float_tag                   10
+#define AllTypes_req_fixed64_tag                 11
+#define AllTypes_req_sfixed64_tag                12
+#define AllTypes_req_double_tag                  13
+#define AllTypes_req_string_tag                  14
+#define AllTypes_req_bytes_tag                   15
+#define AllTypes_req_submsg_tag                  16
+#define AllTypes_req_enum_tag                    17
+#define AllTypes_rep_int32_tag                   21
+#define AllTypes_rep_int64_tag                   22
+#define AllTypes_rep_uint32_tag                  23
+#define AllTypes_rep_uint64_tag                  24
+#define AllTypes_rep_sint32_tag                  25
+#define AllTypes_rep_sint64_tag                  26
+#define AllTypes_rep_bool_tag                    27
+#define AllTypes_rep_fixed32_tag                 28
+#define AllTypes_rep_sfixed32_tag                29
+#define AllTypes_rep_float_tag                   30
+#define AllTypes_rep_fixed64_tag                 31
+#define AllTypes_rep_sfixed64_tag                32
+#define AllTypes_rep_double_tag                  33
+#define AllTypes_rep_string_tag                  34
+#define AllTypes_rep_bytes_tag                   35
+#define AllTypes_rep_submsg_tag                  36
+#define AllTypes_rep_enum_tag                    37
+#define AllTypes_opt_int32_tag                   41
+#define AllTypes_opt_int64_tag                   42
+#define AllTypes_opt_uint32_tag                  43
+#define AllTypes_opt_uint64_tag                  44
+#define AllTypes_opt_sint32_tag                  45
+#define AllTypes_opt_sint64_tag                  46
+#define AllTypes_opt_bool_tag                    47
+#define AllTypes_opt_fixed32_tag                 48
+#define AllTypes_opt_sfixed32_tag                49
+#define AllTypes_opt_float_tag                   50
+#define AllTypes_opt_fixed64_tag                 51
+#define AllTypes_opt_sfixed64_tag                52
+#define AllTypes_opt_double_tag                  53
+#define AllTypes_opt_string_tag                  54
+#define AllTypes_opt_bytes_tag                   55
+#define AllTypes_opt_submsg_tag                  56
+#define AllTypes_opt_enum_tag                    57
+#define AllTypes_end_tag                         99
+
+/* Struct field encoding specification for nanopb */
+extern const pb_field_t SubMessage_fields[4];
+extern const pb_field_t EmptyMessage_fields[1];
+extern const pb_field_t Limits_fields[11];
+extern const pb_field_t AllTypes_fields[54];
+
+/* Maximum encoded size of messages (where known) */
+#define SubMessage_size                          34
+#define EmptyMessage_size                        0
+#define Limits_size                              90
+#define AllTypes_size                            1362
+
+#ifdef __cplusplus
+} /* extern "C" */
+#endif
+
+#endif

tests/backwards_compatibility/alltypes_legacy.options

@@ -0,0 +1,3 @@
+* max_size:16
+* max_count:5
+

tests/backwards_compatibility/alltypes_legacy.proto

@@ -8,6 +8,24 @@ message EmptyMessage {
 
 }
 
+enum HugeEnum {
+    Negative = -2147483647; /* protoc doesn't accept -2147483648 here */
+    Positive =  2147483647;
+}
+
+message Limits {
+    required int32      int32_min  =  1 [default = 2147483647];
+    required int32      int32_max  =  2 [default = -2147483647];
+    required uint32     uint32_min =  3 [default = 4294967295];
+    required uint32     uint32_max =  4 [default = 0];
+    required int64      int64_min  =  5 [default = 9223372036854775807];
+    required int64      int64_max  =  6 [default = -9223372036854775807];
+    required uint64     uint64_min =  7 [default = 18446744073709551615];
+    required uint64     uint64_max =  8 [default = 0];
+    required HugeEnum   enum_min   =  9 [default = Positive];
+    required HugeEnum   enum_max   = 10 [default = Negative];
+}
+
 enum MyEnum {
     Zero = 0;
     First = 1;
@@ -36,30 +54,28 @@ message AllTypes {
     required bytes      req_bytes   = 15;
     required SubMessage req_submsg  = 16;
     required MyEnum     req_enum    = 17;
-    required EmptyMessage req_emptymsg = 18;
     
     
-    repeated int32      rep_int32   = 21;
-    repeated int64      rep_int64   = 22;
-    repeated uint32     rep_uint32  = 23;
-    repeated uint64     rep_uint64  = 24;
-    repeated sint32     rep_sint32  = 25;
-    repeated sint64     rep_sint64  = 26;
-    repeated bool       rep_bool    = 27;
+    repeated int32      rep_int32   = 21 [packed = true];
+    repeated int64      rep_int64   = 22 [packed = true];
+    repeated uint32     rep_uint32  = 23 [packed = true];
+    repeated uint64     rep_uint64  = 24 [packed = true];
+    repeated sint32     rep_sint32  = 25 [packed = true];
+    repeated sint64     rep_sint64  = 26 [packed = true];
+    repeated bool       rep_bool    = 27 [packed = true];
     
-    repeated fixed32    rep_fixed32 = 28;
-    repeated sfixed32   rep_sfixed32= 29;
-    repeated float      rep_float   = 30;
+    repeated fixed32    rep_fixed32 = 28 [packed = true];
+    repeated sfixed32   rep_sfixed32= 29 [packed = true];
+    repeated float      rep_float   = 30 [packed = true];
     
-    repeated fixed64    rep_fixed64 = 31;
-    repeated sfixed64   rep_sfixed64= 32;
-    repeated double     rep_double  = 33;
+    repeated fixed64    rep_fixed64 = 31 [packed = true];
+    repeated sfixed64   rep_sfixed64= 32 [packed = true];
+    repeated double     rep_double  = 33 [packed = true];
     
     repeated string     rep_string  = 34;
     repeated bytes      rep_bytes   = 35;
     repeated SubMessage rep_submsg  = 36;
-    repeated MyEnum     rep_enum    = 37;
-    repeated EmptyMessage rep_emptymsg = 38;
+    repeated MyEnum     rep_enum    = 37 [packed = true];
     
     optional int32      opt_int32   = 41 [default = 4041];
     optional int64      opt_int64   = 42 [default = 4042];
@@ -81,7 +97,6 @@ message AllTypes {
     optional bytes      opt_bytes   = 55 [default = "4055"];
     optional SubMessage opt_submsg  = 56;
     optional MyEnum     opt_enum    = 57 [default = Second];
-    optional EmptyMessage opt_emptymsg = 58;
 
     // Just to make sure that the size of the fields has been calculated
     // properly, i.e. otherwise a bug in last field might not be detected.

tests/backwards_compatibility/decode_legacy.c

@@ -1,16 +1,17 @@
 /* Tests the decoding of all types.
- * This is a backwards-compatibility test, using bc_alltypes.pb.h.
- * It is similar to test_decode3, but duplicated in order to allow
- * test_decode3 to test any new features introduced later.
+ * This is a backwards-compatibility test, using alltypes_legacy.h.
+ * It is similar to decode_alltypes, but duplicated in order to allow
+ * decode_alltypes to test any new features introduced later.
  *
- * Run e.g. ./bc_encode | ./bc_decode
+ * Run e.g. ./encode_legacy | ./decode_legacy
  */
 
 #include <stdio.h>
 #include <string.h>
 #include <stdlib.h>
 #include <pb_decode.h>
-#include "bc_alltypes.pb.h"
+#include "alltypes_legacy.h"
+#include "test_helpers.h"
 
 #define TEST(x) if (!(x)) { \
     printf("Test " #x " failed.\n"); \
@@ -176,15 +177,19 @@ bool check_alltypes(pb_istream_t *stream, int mode)
 
 int main(int argc, char **argv)
 {
+    uint8_t buffer[1024];
+    size_t count;
+    pb_istream_t stream;
+
     /* Whether to expect the optional values or the default values. */
     int mode = (argc > 1) ? atoi(argv[1]) : 0;
     
     /* Read the data into buffer */
-    uint8_t buffer[1024];
-    size_t count = fread(buffer, 1, sizeof(buffer), stdin);
+    SET_BINARY_MODE(stdin);
+    count = fread(buffer, 1, sizeof(buffer), stdin);
     
     /* Construct a pb_istream_t for reading from the buffer */
-    pb_istream_t stream = pb_istream_from_buffer(buffer, count);
+    stream = pb_istream_from_buffer(buffer, count);
     
     /* Decode and print out the stuff */
     if (!check_alltypes(&stream, mode))

tests/backwards_compatibility/encode_legacy.c

@@ -1,14 +1,15 @@
 /* Attempts to test all the datatypes supported by ProtoBuf.
- * This is a backwards-compatibility test, using bc_alltypes.pb.h.
- * It is similar to test_encode3, but duplicated in order to allow
- * test_encode3 to test any new features introduced later.
+ * This is a backwards-compatibility test, using alltypes_legacy.h.
+ * It is similar to encode_alltypes, but duplicated in order to allow
+ * encode_alltypes to test any new features introduced later.
  */
 
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
 #include <pb_encode.h>
-#include "bc_alltypes.pb.h"
+#include "alltypes_legacy.h"
+#include "test_helpers.h"
 
 int main(int argc, char **argv)
 {
@@ -113,19 +114,22 @@ int main(int argc, char **argv)
     }
     
     alltypes.end = 1099;
-    
-    uint8_t buffer[1024];
-    pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
-    
-    /* Now encode it and check if we succeeded. */
-    if (pb_encode(&stream, AllTypes_fields, &alltypes))
-    {
-        fwrite(buffer, 1, stream.bytes_written, stdout);
-        return 0; /* Success */
-    }
-    else
-    {
-        fprintf(stderr, "Encoding failed!\n");
-        return 1; /* Failure */
+
+    {    
+        uint8_t buffer[1024];
+        pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
+        
+        /* Now encode it and check if we succeeded. */
+        if (pb_encode(&stream, AllTypes_fields, &alltypes))
+        {
+            SET_BINARY_MODE(stdout);
+            fwrite(buffer, 1, stream.bytes_written, stdout);
+            return 0; /* Success */
+        }
+        else
+        {
+            fprintf(stderr, "Encoding failed!\n");
+            return 1; /* Failure */
+        }
     }
 }

tests/basic_buffer/SConscript

@@ -0,0 +1,12 @@
+# Build and run a basic round-trip test using memory buffer encoding.
+
+Import("env")
+
+enc = env.Program(["encode_buffer.c", "$COMMON/person.pb.c", "$COMMON/pb_encode.o", "$COMMON/pb_common.o"])
+dec = env.Program(["decode_buffer.c", "$COMMON/person.pb.c", "$COMMON/pb_decode.o", "$COMMON/pb_common.o"])
+
+env.RunTest(enc)
+env.RunTest([dec, "encode_buffer.output"])
+env.Decode(["encode_buffer.output", "$COMMON/person.proto"], MESSAGE = "Person")
+env.Compare(["decode_buffer.output", "encode_buffer.decoded"])
+

tests/basic_buffer/decode_buffer.c

@@ -9,13 +9,14 @@
 #include <stdio.h>
 #include <pb_decode.h>
 #include "person.pb.h"
+#include "test_helpers.h"
 
 /* This function is called once from main(), it handles
    the decoding and printing. */
 bool print_person(pb_istream_t *stream)
 {
     int i;
-    Person person;
+    Person person = Person_init_zero;
     
     if (!pb_decode(stream, Person_fields, &person))
         return false;
@@ -59,9 +60,13 @@ bool print_person(pb_istream_t *stream)
 
 int main()
 {
+    uint8_t buffer[Person_size];
+    pb_istream_t stream;
+    size_t count;
+    
     /* Read the data into buffer */
-    uint8_t buffer[512];
-    size_t count = fread(buffer, 1, sizeof(buffer), stdin);
+    SET_BINARY_MODE(stdin);
+    count = fread(buffer, 1, sizeof(buffer), stdin);
     
     if (!feof(stdin))
     {
@@ -70,7 +75,7 @@ int main()
     }
     
     /* Construct a pb_istream_t for reading from the buffer */
-    pb_istream_t stream = pb_istream_from_buffer(buffer, count);
+    stream = pb_istream_from_buffer(buffer, count);
     
     /* Decode and print out the stuff */
     if (!print_person(&stream))

tests/basic_buffer/encode_buffer.c

@@ -6,9 +6,13 @@
 #include <stdio.h>
 #include <pb_encode.h>
 #include "person.pb.h"
+#include "test_helpers.h"
 
 int main()
 {
+    uint8_t buffer[Person_size];
+    pb_ostream_t stream;
+    
     /* Initialize the structure with constants */
     Person person = {"Test Person 99", 99, true, "test@person.com",
         3, {{"555-12345678", true, Person_PhoneType_MOBILE},
@@ -16,12 +20,13 @@ int main()
             {"1234-5678", true, Person_PhoneType_WORK},
         }};
 
-    uint8_t buffer[512];
-    pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
+    stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
     
     /* Now encode it and check if we succeeded. */
     if (pb_encode(&stream, Person_fields, &person))
-    {
+    {    
+        /* Write the result data to stdout */
+        SET_BINARY_MODE(stdout);
         fwrite(buffer, 1, stream.bytes_written, stdout);
         return 0; /* Success */
     }

tests/basic_stream/SConscript

@@ -0,0 +1,12 @@
+# Build and run a basic round-trip test using direct stream encoding.
+
+Import("env")
+
+enc = env.Program(["encode_stream.c", "$COMMON/person.pb.c", "$COMMON/pb_encode.o", "$COMMON/pb_common.o"])
+dec = env.Program(["decode_stream.c", "$COMMON/person.pb.c", "$COMMON/pb_decode.o", "$COMMON/pb_common.o"])
+
+env.RunTest(enc)
+env.RunTest([dec, "encode_stream.output"])
+env.Decode(["encode_stream.output", "$COMMON/person.proto"], MESSAGE = "Person")
+env.Compare(["decode_stream.output", "encode_stream.decoded"])
+

tests/basic_stream/decode_stream.c

@@ -4,6 +4,7 @@
 #include <stdio.h>
 #include <pb_decode.h>
 #include "person.pb.h"
+#include "test_helpers.h"
 
 /* This function is called once from main(), it handles
    the decoding and printing.
@@ -11,7 +12,7 @@
 bool print_person(pb_istream_t *stream)
 {
     int i;
-    Person person;
+    Person person = Person_init_zero;
     
     if (!pb_decode(stream, Person_fields, &person))
         return false;
@@ -69,10 +70,10 @@ bool callback(pb_istream_t *stream, uint8_t *buf, size_t count)
 
 int main()
 {
-    /* Maximum size is specified to prevent infinite length messages from
-     * hanging this in the fuzz test.
-     */
-    pb_istream_t stream = {&callback, stdin, 10000};
+    pb_istream_t stream = {&callback, NULL, SIZE_MAX};
+    stream.state = stdin;
+    SET_BINARY_MODE(stdin);
+
     if (!print_person(&stream))
     {
         printf("Parsing failed: %s\n", PB_GET_ERROR(&stream));

tests/basic_stream/encode_stream.c

@@ -4,6 +4,7 @@
 #include <stdio.h>
 #include <pb_encode.h>
 #include "person.pb.h"
+#include "test_helpers.h"
 
 /* This binds the pb_ostream_t into the stdout stream */
 bool streamcallback(pb_ostream_t *stream, const uint8_t *buf, size_t count)
@@ -22,7 +23,9 @@ int main()
         }};
     
     /* Prepare the stream, output goes directly to stdout */
-    pb_ostream_t stream = {&streamcallback, stdout, SIZE_MAX, 0};
+    pb_ostream_t stream = {&streamcallback, NULL, SIZE_MAX, 0};
+    stream.state = stdout;
+    SET_BINARY_MODE(stdout);
     
     /* Now encode it and check if we succeeded. */
     if (pb_encode(&stream, Person_fields, &person))

tests/bc_alltypes.pb.c

@@ -1,93 +0,0 @@
-/* Automatically generated nanopb constant definitions */
-/* Generated by 0.2.0-dev at Sun Feb 17 00:09:53 2013. */
-/* This is a file generated using nanopb-0.2.0-dev.
- * It is used as a part of test suite in order to detect any
- * incompatible changes made to the generator in future versions.
- */
-
-#include "bc_alltypes.pb.h"
-
-const char SubMessage_substuff1_default[16] = "1";
-const int32_t SubMessage_substuff2_default = 2;
-const uint32_t SubMessage_substuff3_default = 3;
-const int32_t AllTypes_opt_int32_default = 4041;
-const int64_t AllTypes_opt_int64_default = 4042;
-const uint32_t AllTypes_opt_uint32_default = 4043;
-const uint64_t AllTypes_opt_uint64_default = 4044;
-const int32_t AllTypes_opt_sint32_default = 4045;
-const int64_t AllTypes_opt_sint64_default = 4046;
-const bool AllTypes_opt_bool_default = false;
-const uint32_t AllTypes_opt_fixed32_default = 4048;
-const int32_t AllTypes_opt_sfixed32_default = 4049;
-const float AllTypes_opt_float_default = 4050;
-const uint64_t AllTypes_opt_fixed64_default = 4051;
-const int64_t AllTypes_opt_sfixed64_default = 4052;
-const double AllTypes_opt_double_default = 4053;
-const char AllTypes_opt_string_default[16] = "4054";
-const AllTypes_opt_bytes_t AllTypes_opt_bytes_default = {4, {0x34,0x30,0x35,0x35}};
-const MyEnum AllTypes_opt_enum_default = MyEnum_Second;
-
-
-const pb_field_t SubMessage_fields[4] = {
-    PB_FIELD(  1, STRING  , REQUIRED, STATIC, SubMessage, substuff1, substuff1, &SubMessage_substuff1_default),
-    PB_FIELD(  2, INT32   , REQUIRED, STATIC, SubMessage, substuff2, substuff1, &SubMessage_substuff2_default),
-    PB_FIELD(  3, FIXED32 , OPTIONAL, STATIC, SubMessage, substuff3, substuff2, &SubMessage_substuff3_default),
-    PB_LAST_FIELD
-};
-
-const pb_field_t AllTypes_fields[53] = {
-    PB_FIELD(  1, INT32   , REQUIRED, STATIC, AllTypes, req_int32, req_int32, 0),
-    PB_FIELD(  2, INT64   , REQUIRED, STATIC, AllTypes, req_int64, req_int32, 0),
-    PB_FIELD(  3, UINT32  , REQUIRED, STATIC, AllTypes, req_uint32, req_int64, 0),
-    PB_FIELD(  4, UINT64  , REQUIRED, STATIC, AllTypes, req_uint64, req_uint32, 0),
-    PB_FIELD(  5, SINT32  , REQUIRED, STATIC, AllTypes, req_sint32, req_uint64, 0),
-    PB_FIELD(  6, SINT64  , REQUIRED, STATIC, AllTypes, req_sint64, req_sint32, 0),
-    PB_FIELD(  7, BOOL    , REQUIRED, STATIC, AllTypes, req_bool, req_sint64, 0),
-    PB_FIELD(  8, FIXED32 , REQUIRED, STATIC, AllTypes, req_fixed32, req_bool, 0),
-    PB_FIELD(  9, SFIXED32, REQUIRED, STATIC, AllTypes, req_sfixed32, req_fixed32, 0),
-    PB_FIELD( 10, FLOAT   , REQUIRED, STATIC, AllTypes, req_float, req_sfixed32, 0),
-    PB_FIELD( 11, FIXED64 , REQUIRED, STATIC, AllTypes, req_fixed64, req_float, 0),
-    PB_FIELD( 12, SFIXED64, REQUIRED, STATIC, AllTypes, req_sfixed64, req_fixed64, 0),
-    PB_FIELD( 13, DOUBLE  , REQUIRED, STATIC, AllTypes, req_double, req_sfixed64, 0),
-    PB_FIELD( 14, STRING  , REQUIRED, STATIC, AllTypes, req_string, req_double, 0),
-    PB_FIELD( 15, BYTES   , REQUIRED, STATIC, AllTypes, req_bytes, req_string, 0),
-    PB_FIELD( 16, MESSAGE , REQUIRED, STATIC, AllTypes, req_submsg, req_bytes, &SubMessage_fields),
-    PB_FIELD( 17, ENUM    , REQUIRED, STATIC, AllTypes, req_enum, req_submsg, 0),
-    PB_FIELD( 21, INT32   , REPEATED, STATIC, AllTypes, rep_int32, req_enum, 0),
-    PB_FIELD( 22, INT64   , REPEATED, STATIC, AllTypes, rep_int64, rep_int32, 0),
-    PB_FIELD( 23, UINT32  , REPEATED, STATIC, AllTypes, rep_uint32, rep_int64, 0),
-    PB_FIELD( 24, UINT64  , REPEATED, STATIC, AllTypes, rep_uint64, rep_uint32, 0),
-    PB_FIELD( 25, SINT32  , REPEATED, STATIC, AllTypes, rep_sint32, rep_uint64, 0),
-    PB_FIELD( 26, SINT64  , REPEATED, STATIC, AllTypes, rep_sint64, rep_sint32, 0),
-    PB_FIELD( 27, BOOL    , REPEATED, STATIC, AllTypes, rep_bool, rep_sint64, 0),
-    PB_FIELD( 28, FIXED32 , REPEATED, STATIC, AllTypes, rep_fixed32, rep_bool, 0),
-    PB_FIELD( 29, SFIXED32, REPEATED, STATIC, AllTypes, rep_sfixed32, rep_fixed32, 0),
-    PB_FIELD( 30, FLOAT   , REPEATED, STATIC, AllTypes, rep_float, rep_sfixed32, 0),
-    PB_FIELD( 31, FIXED64 , REPEATED, STATIC, AllTypes, rep_fixed64, rep_float, 0),
-    PB_FIELD( 32, SFIXED64, REPEATED, STATIC, AllTypes, rep_sfixed64, rep_fixed64, 0),
-    PB_FIELD( 33, DOUBLE  , REPEATED, STATIC, AllTypes, rep_double, rep_sfixed64, 0),
-    PB_FIELD( 34, STRING  , REPEATED, STATIC, AllTypes, rep_string, rep_double, 0),
-    PB_FIELD( 35, BYTES   , REPEATED, STATIC, AllTypes, rep_bytes, rep_string, 0),
-    PB_FIELD( 36, MESSAGE , REPEATED, STATIC, AllTypes, rep_submsg, rep_bytes, &SubMessage_fields),
-    PB_FIELD( 37, ENUM    , REPEATED, STATIC, AllTypes, rep_enum, rep_submsg, 0),
-    PB_FIELD( 41, INT32   , OPTIONAL, STATIC, AllTypes, opt_int32, rep_enum, &AllTypes_opt_int32_default),
-    PB_FIELD( 42, INT64   , OPTIONAL, STATIC, AllTypes, opt_int64, opt_int32, &AllTypes_opt_int64_default),
-    PB_FIELD( 43, UINT32  , OPTIONAL, STATIC, AllTypes, opt_uint32, opt_int64, &AllTypes_opt_uint32_default),
-    PB_FIELD( 44, UINT64  , OPTIONAL, STATIC, AllTypes, opt_uint64, opt_uint32, &AllTypes_opt_uint64_default),
-    PB_FIELD( 45, SINT32  , OPTIONAL, STATIC, AllTypes, opt_sint32, opt_uint64, &AllTypes_opt_sint32_default),
-    PB_FIELD( 46, SINT64  , OPTIONAL, STATIC, AllTypes, opt_sint64, opt_sint32, &AllTypes_opt_sint64_default),
-    PB_FIELD( 47, BOOL    , OPTIONAL, STATIC, AllTypes, opt_bool, opt_sint64, &AllTypes_opt_bool_default),
-    PB_FIELD( 48, FIXED32 , OPTIONAL, STATIC, AllTypes, opt_fixed32, opt_bool, &AllTypes_opt_fixed32_default),
-    PB_FIELD( 49, SFIXED32, OPTIONAL, STATIC, AllTypes, opt_sfixed32, opt_fixed32, &AllTypes_opt_sfixed32_default),
-    PB_FIELD( 50, FLOAT   , OPTIONAL, STATIC, AllTypes, opt_float, opt_sfixed32, &AllTypes_opt_float_default),
-    PB_FIELD( 51, FIXED64 , OPTIONAL, STATIC, AllTypes, opt_fixed64, opt_float, &AllTypes_opt_fixed64_default),
-    PB_FIELD( 52, SFIXED64, OPTIONAL, STATIC, AllTypes, opt_sfixed64, opt_fixed64, &AllTypes_opt_sfixed64_default),
-    PB_FIELD( 53, DOUBLE  , OPTIONAL, STATIC, AllTypes, opt_double, opt_sfixed64, &AllTypes_opt_double_default),
-    PB_FIELD( 54, STRING  , OPTIONAL, STATIC, AllTypes, opt_string, opt_double, &AllTypes_opt_string_default),
-    PB_FIELD( 55, BYTES   , OPTIONAL, STATIC, AllTypes, opt_bytes, opt_string, &AllTypes_opt_bytes_default),
-    PB_FIELD( 56, MESSAGE , OPTIONAL, STATIC, AllTypes, opt_submsg, opt_bytes, &SubMessage_fields),
-    PB_FIELD( 57, ENUM    , OPTIONAL, STATIC, AllTypes, opt_enum, opt_submsg, &AllTypes_opt_enum_default),
-    PB_FIELD( 99, INT32   , REQUIRED, STATIC, AllTypes, end, opt_enum, 0),
-    PB_LAST_FIELD
-};
-

tests/bc_alltypes.pb.h

@@ -1,178 +0,0 @@
-/* Automatically generated nanopb header */
-/* This is a file generated using nanopb-0.2.0-dev.
- * It is used as a part of test suite in order to detect any
- * incompatible changes made to the generator in future versions.
- */
-
-#ifndef _PB_ALLTYPES_PB_H_
-#define _PB_ALLTYPES_PB_H_
-#include <pb.h>
-
-#ifdef __cplusplus
-extern "C" {
-#endif
-
-/* Enum definitions */
-typedef enum _MyEnum {
-    MyEnum_Zero = 0,
-    MyEnum_First = 1,
-    MyEnum_Second = 2,
-    MyEnum_Truth = 42
-} MyEnum;
-
-/* Struct definitions */
-typedef struct _SubMessage {
-    char substuff1[16];
-    int32_t substuff2;
-    bool has_substuff3;
-    uint32_t substuff3;
-} SubMessage;
-
-typedef struct {
-    size_t size;
-    uint8_t bytes[16];
-} AllTypes_req_bytes_t;
-
-typedef struct {
-    size_t size;
-    uint8_t bytes[16];
-} AllTypes_rep_bytes_t;
-
-typedef struct {
-    size_t size;
-    uint8_t bytes[16];
-} AllTypes_opt_bytes_t;
-
-typedef struct _AllTypes {
-    int32_t req_int32;
-    int64_t req_int64;
-    uint32_t req_uint32;
-    uint64_t req_uint64;
-    int32_t req_sint32;
-    int64_t req_sint64;
-    bool req_bool;
-    uint32_t req_fixed32;
-    int32_t req_sfixed32;
-    float req_float;
-    uint64_t req_fixed64;
-    int64_t req_sfixed64;
-    double req_double;
-    char req_string[16];
-    AllTypes_req_bytes_t req_bytes;
-    SubMessage req_submsg;
-    MyEnum req_enum;
-    size_t rep_int32_count;
-    int32_t rep_int32[5];
-    size_t rep_int64_count;
-    int64_t rep_int64[5];
-    size_t rep_uint32_count;
-    uint32_t rep_uint32[5];
-    size_t rep_uint64_count;
-    uint64_t rep_uint64[5];
-    size_t rep_sint32_count;
-    int32_t rep_sint32[5];
-    size_t rep_sint64_count;
-    int64_t rep_sint64[5];
-    size_t rep_bool_count;
-    bool rep_bool[5];
-    size_t rep_fixed32_count;
-    uint32_t rep_fixed32[5];
-    size_t rep_sfixed32_count;
-    int32_t rep_sfixed32[5];
-    size_t rep_float_count;
-    float rep_float[5];
-    size_t rep_fixed64_count;
-    uint64_t rep_fixed64[5];
-    size_t rep_sfixed64_count;
-    int64_t rep_sfixed64[5];
-    size_t rep_double_count;
-    double rep_double[5];
-    size_t rep_string_count;
-    char rep_string[5][16];
-    size_t rep_bytes_count;
-    AllTypes_rep_bytes_t rep_bytes[5];
-    size_t rep_submsg_count;
-    SubMessage rep_submsg[5];
-    size_t rep_enum_count;
-    MyEnum rep_enum[5];
-    bool has_opt_int32;
-    int32_t opt_int32;
-    bool has_opt_int64;
-    int64_t opt_int64;
-    bool has_opt_uint32;
-    uint32_t opt_uint32;
-    bool has_opt_uint64;
-    uint64_t opt_uint64;
-    bool has_opt_sint32;
-    int32_t opt_sint32;
-    bool has_opt_sint64;
-    int64_t opt_sint64;
-    bool has_opt_bool;
-    bool opt_bool;
-    bool has_opt_fixed32;
-    uint32_t opt_fixed32;
-    bool has_opt_sfixed32;
-    int32_t opt_sfixed32;
-    bool has_opt_float;
-    float opt_float;
-    bool has_opt_fixed64;
-    uint64_t opt_fixed64;
-    bool has_opt_sfixed64;
-    int64_t opt_sfixed64;
-    bool has_opt_double;
-    double opt_double;
-    bool has_opt_string;
-    char opt_string[16];
-    bool has_opt_bytes;
-    AllTypes_opt_bytes_t opt_bytes;
-    bool has_opt_submsg;
-    SubMessage opt_submsg;
-    bool has_opt_enum;
-    MyEnum opt_enum;
-    int32_t end;
-} AllTypes;
-
-/* Default values for struct fields */
-extern const char SubMessage_substuff1_default[16];
-extern const int32_t SubMessage_substuff2_default;
-extern const uint32_t SubMessage_substuff3_default;
-extern const int32_t AllTypes_opt_int32_default;
-extern const int64_t AllTypes_opt_int64_default;
-extern const uint32_t AllTypes_opt_uint32_default;
-extern const uint64_t AllTypes_opt_uint64_default;
-extern const int32_t AllTypes_opt_sint32_default;
-extern const int64_t AllTypes_opt_sint64_default;
-extern const bool AllTypes_opt_bool_default;
-extern const uint32_t AllTypes_opt_fixed32_default;
-extern const int32_t AllTypes_opt_sfixed32_default;
-extern const float AllTypes_opt_float_default;
-extern const uint64_t AllTypes_opt_fixed64_default;
-extern const int64_t AllTypes_opt_sfixed64_default;
-extern const double AllTypes_opt_double_default;
-extern const char AllTypes_opt_string_default[16];
-extern const AllTypes_opt_bytes_t AllTypes_opt_bytes_default;
-extern const MyEnum AllTypes_opt_enum_default;
-
-/* Struct field encoding specification for nanopb */
-extern const pb_field_t SubMessage_fields[4];
-extern const pb_field_t AllTypes_fields[53];
-
-/* Check that field information fits in pb_field_t */
-#if !defined(PB_FIELD_16BIT) && !defined(PB_FIELD_32BIT)
-STATIC_ASSERT((pb_membersize(AllTypes, req_submsg) < 256 && pb_membersize(AllTypes, rep_submsg[0]) < 256 && pb_membersize(AllTypes, opt_submsg) < 256), YOU_MUST_DEFINE_PB_FIELD_16BIT_FOR_MESSAGES_SubMessage_AllTypes)
-#endif
-
-#if !defined(PB_FIELD_32BIT)
-STATIC_ASSERT((pb_membersize(AllTypes, req_submsg) < 65536 && pb_membersize(AllTypes, rep_submsg[0]) < 65536 && pb_membersize(AllTypes, opt_submsg) < 65536), YOU_MUST_DEFINE_PB_FIELD_32BIT_FOR_MESSAGES_SubMessage_AllTypes)
-#endif
-
-/* On some platforms (such as AVR), double is really float.
- * These are not directly supported by nanopb, but see example_avr_double.
- */
-STATIC_ASSERT(sizeof(double) == 8, DOUBLE_MUST_BE_8_BYTES)
-
-#ifdef __cplusplus
-} /* extern "C" */
-#endif
-
-#endif

tests/buffer_only/SConscript

@@ -0,0 +1,28 @@
+# Run the alltypes test case, but compile with PB_BUFFER_ONLY=1
+
+Import("env")
+
+# Take copy of the files for custom build.
+c = Copy("$TARGET", "$SOURCE")
+env.Command("alltypes.pb.h", "$BUILD/alltypes/alltypes.pb.h", c)
+env.Command("alltypes.pb.c", "$BUILD/alltypes/alltypes.pb.c", c)
+env.Command("encode_alltypes.c", "$BUILD/alltypes/encode_alltypes.c", c)
+env.Command("decode_alltypes.c", "$BUILD/alltypes/decode_alltypes.c", c)
+
+# Define the compilation options
+opts = env.Clone()
+opts.Append(CPPDEFINES = {'PB_BUFFER_ONLY': 1})
+
+# Build new version of core
+strict = opts.Clone()
+strict.Append(CFLAGS = strict['CORECFLAGS'])
+strict.Object("pb_decode_bufonly.o", "$NANOPB/pb_decode.c")
+strict.Object("pb_encode_bufonly.o", "$NANOPB/pb_encode.c")
+strict.Object("pb_common_bufonly.o", "$NANOPB/pb_common.c")
+
+# Now build and run the test normally.
+enc = opts.Program(["encode_alltypes.c", "alltypes.pb.c", "pb_encode_bufonly.o", "pb_common_bufonly.o"])
+dec = opts.Program(["decode_alltypes.c", "alltypes.pb.c", "pb_decode_bufonly.o", "pb_common_bufonly.o"])
+
+env.RunTest(enc)
+env.RunTest([dec, "encode_alltypes.output"])

tests/callbacks/SConscript

@@ -0,0 +1,14 @@
+# Test the functionality of the callback fields.
+
+Import("env")
+
+env.NanopbProto("callbacks")
+enc = env.Program(["encode_callbacks.c", "callbacks.pb.c", "$COMMON/pb_encode.o", "$COMMON/pb_common.o"])
+dec = env.Program(["decode_callbacks.c", "callbacks.pb.c", "$COMMON/pb_decode.o", "$COMMON/pb_common.o"])
+
+env.RunTest(enc)
+env.RunTest([dec, "encode_callbacks.output"])
+
+env.Decode(["encode_callbacks.output", "callbacks.proto"], MESSAGE = "TestMessage")
+env.Compare(["decode_callbacks.output", "encode_callbacks.decoded"])
+

tests/callbacks/decode_callbacks.c

@@ -5,6 +5,7 @@
 #include <stdio.h>
 #include <pb_decode.h>
 #include "callbacks.pb.h"
+#include "test_helpers.h"
 
 bool print_string(pb_istream_t *stream, const pb_field_t *field, void **arg)
 {
@@ -50,21 +51,24 @@ bool print_fixed64(pb_istream_t *stream, const pb_field_t *field, void **arg)
     if (!pb_decode_fixed64(stream, &value))
         return false;
     
-    printf((char*)*arg, (long long)value);
+    printf((char*)*arg, (long)value);
     return true;
 }
 
 int main()
 {
     uint8_t buffer[1024];
-    size_t length = fread(buffer, 1, 1024, stdin);
-    pb_istream_t stream = pb_istream_from_buffer(buffer, length);
-    
+    size_t length;
+    pb_istream_t stream;
     /* Note: empty initializer list initializes the struct with all-0.
      * This is recommended so that unused callbacks are set to NULL instead
      * of crashing at runtime.
      */
-    TestMessage testmessage = {};
+    TestMessage testmessage = {{{NULL}}};
+    
+    SET_BINARY_MODE(stdin);
+    length = fread(buffer, 1, 1024, stdin);
+    stream = pb_istream_from_buffer(buffer, length);    
     
     testmessage.submsg.stringvalue.funcs.decode = &print_string;
     testmessage.submsg.stringvalue.arg = "submsg {\n  stringvalue: \"%s\"\n";
@@ -73,7 +77,7 @@ int main()
     testmessage.submsg.fixed32value.funcs.decode = &print_fixed32;
     testmessage.submsg.fixed32value.arg = "  fixed32value: %ld\n";
     testmessage.submsg.fixed64value.funcs.decode = &print_fixed64;
-    testmessage.submsg.fixed64value.arg = "  fixed64value: %lld\n}\n";
+    testmessage.submsg.fixed64value.arg = "  fixed64value: %ld\n}\n";
     
     testmessage.stringvalue.funcs.decode = &print_string;
     testmessage.stringvalue.arg = "stringvalue: \"%s\"\n";
@@ -82,7 +86,7 @@ int main()
     testmessage.fixed32value.funcs.decode = &print_fixed32;
     testmessage.fixed32value.arg = "fixed32value: %ld\n";
     testmessage.fixed64value.funcs.decode = &print_fixed64;
-    testmessage.fixed64value.arg = "fixed64value: %lld\n";
+    testmessage.fixed64value.arg = "fixed64value: %ld\n";
     testmessage.repeatedstring.funcs.decode = &print_string;
     testmessage.repeatedstring.arg = "repeatedstring: \"%s\"\n";
     

tests/callbacks/encode_callbacks.c

@@ -4,6 +4,7 @@
 #include <string.h>
 #include <pb_encode.h>
 #include "callbacks.pb.h"
+#include "test_helpers.h"
 
 bool encode_string(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
 {
@@ -25,19 +26,21 @@ bool encode_int32(pb_ostream_t *stream, const pb_field_t *field, void * const *a
 
 bool encode_fixed32(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
 {
+    uint32_t value = 42;
+
     if (!pb_encode_tag_for_field(stream, field))
         return false;
     
-    uint32_t value = 42;
     return pb_encode_fixed32(stream, &value);
 }
 
 bool encode_fixed64(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
 {
+    uint64_t value = 42;
+
     if (!pb_encode_tag_for_field(stream, field))
         return false;
     
-    uint64_t value = 42;
     return pb_encode_fixed64(stream, &value);
 }
 
@@ -60,8 +63,10 @@ bool encode_repeatedstring(pb_ostream_t *stream, const pb_field_t *field, void *
 int main()
 {
     uint8_t buffer[1024];
-    pb_ostream_t stream = pb_ostream_from_buffer(buffer, 1024);
-    TestMessage testmessage = {};
+    pb_ostream_t stream;
+    TestMessage testmessage = {{{NULL}}};
+    
+    stream = pb_ostream_from_buffer(buffer, 1024);
     
     testmessage.stringvalue.funcs.encode = &encode_string;
     testmessage.int32value.funcs.encode = &encode_int32;
@@ -79,6 +84,7 @@ int main()
     if (!pb_encode(&stream, TestMessage_fields, &testmessage))
         return 1;
     
+    SET_BINARY_MODE(stdout);
     if (fwrite(buffer, stream.bytes_written, 1, stdout) != 1)
         return 2;
     

tests/common/SConscript

@@ -0,0 +1,17 @@
+# Build the common files needed by multiple test cases
+
+Import('env')
+
+# Protocol definitions for the encode/decode_unittests
+env.NanopbProto("unittestproto")
+
+# Protocol definitions for basic_buffer/stream tests
+env.NanopbProto("person")
+
+# Binaries of the pb_decode.c and pb_encode.c
+# These are built using more strict warning flags.
+strict = env.Clone()
+strict.Append(CFLAGS = strict['CORECFLAGS'])
+strict.Object("pb_decode.o", "$NANOPB/pb_decode.c")
+strict.Object("pb_encode.o", "$NANOPB/pb_encode.c")
+strict.Object("pb_common.o", "$NANOPB/pb_common.c")

tests/common/test_helpers.h

@@ -0,0 +1,17 @@
+/* Compatibility helpers for the test programs. */
+
+#ifndef _TEST_HELPERS_H_
+#define _TEST_HELPERS_H_
+
+#ifdef _WIN32
+#include <io.h>
+#include <fcntl.h>
+#define SET_BINARY_MODE(file) setmode(fileno(file), O_BINARY)
+
+#else
+#define SET_BINARY_MODE(file)
+
+#endif
+
+
+#endif

tests/common/unittestproto.proto

@@ -12,6 +12,10 @@ message StringMessage {
     required string data = 1 [(nanopb).max_size = 10];
 }
 
+message BytesMessage {
+    required bytes data = 1 [(nanopb).max_size = 16];
+}
+
 message CallbackArray {
     // We cheat a bit and use this message for testing other types, too.
     // Nanopb does not care about the actual defined data type for callback
@@ -30,3 +34,8 @@ message CallbackContainer {
 message CallbackContainerContainer {
     required CallbackContainer submsg = 1;
 }
+
+message StringPointerContainer {
+    repeated string rep_str = 1 [(nanopb).type = FT_POINTER];
+}
+