Publishing nanopb-0.2.9.2

Fix build failure due to missing dependency in SConscript
Backport memory leak fix for issue 138.
2015-01-24 17:18:25 +02:00 · 2014-12-27 00:38:22 +02:00 · 2014-12-27 00:01:45 +02:00 · 2014-12-26 23:59:56 +02:00 · 2014-09-11 19:14:45 +03:00 · 2014-09-11 19:13:59 +03:00
66 changed files with 3136 additions and 287 deletions
--- a/CHANGELOG.txt
+++ b/CHANGELOG.txt
@@ -1,3 +1,47 @@
+nanopb-0.2.9.2 (2015-01-24)
+ Fix memory leaks with PB_ENABLE_MALLOC with some submessage hierarchies (issue 138)
+ Fix compilation error with generated initializers for repeated pointer fields
+
+nanopb-0.2.9.1 (2014-09-11)
+ Fix security issue due to size_t overflows. (issue 132)
+ Fix memory leak with duplicated fields and PB_ENABLE_MALLOC
+ Fix crash if pb_release() is called twice.
+
+nanopb-0.2.9 (2014-08-09)
+ NOTE: If you are using the -e option with the generator, you have
+ to prepend . to the argument to get the same behaviour as before.
+
+ Do not automatically add a dot with generator -e option. (issue 122)
+ Fix problem with .options file and extension fields. (issue 125)
+ Don't use SIZE_MAX macro, as it is not in C89. (issue 120)
+ Generate #defines for initializing message structures. (issue 79)
+ Add skip_message option to generator. (issue 121)
+ Add PB_PACKED_STRUCT support for Keil MDK-ARM toolchain (issue 119)
+ Give better messages about the .options file path. (issue 124)
+ Improved tests
+
+nanopb-0.2.8 (2014-05-20)
+ Fix security issue with PB_ENABLE_MALLOC. (issue 117)
+ Add option to not add timestamps to .pb.h and .pb.c preambles. (issue 115)
+ Documentation updates
+ Improved tests
+
+nanopb-0.2.7 (2014-04-07)
+ Fix bug with default values for extension fields (issue 111)
+ Fix some MISRA-C warnings (issue 91)
+ Implemented optional malloc() support (issue 80)
+ Changed pointer-type bytes field datatype
+ Add a "found" field to pb_extension_t (issue 112)
+ Add convenience function pb_get_encoded_size() (issue 16)
+
+nanopb-0.2.6 (2014-02-15)
+ Fix generator error with bytes callback fields (issue 99)
+ Fix warnings about large integer constants (issue 102)
+ Add comments to where STATIC_ASSERT is used (issue 96)
+ Add warning about unknown field names on .options (issue 105)
+ Move descriptor.proto to google/protobuf subdirectory (issue 104)
+ Improved tests
+
 nanopb-0.2.5 (2014-01-01)
 Fix a bug with encoding negative values in int32 fields (issue 97)
 Create binary packages of the generator + dependencies (issue 47)
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -47,7 +47,6 @@ Features and limitations

 **Limitations**

-#) User must provide callbacks when decoding arrays or strings without maximum size. Malloc support could be added as a separate module.
 #) Some speed has been sacrificed for code size.
 #) Encoding is focused on writing to streams. For memory buffers only it could be made more efficient.
 #) The deprecated Protocol Buffers feature called "groups" is not supported.
--- a/docs/reference.rst
+++ b/docs/reference.rst
@@ -24,10 +24,8 @@ __BIG_ENDIAN__                 Set this if your platform stores integers and
                               floats in big-endian format. Mixed-endian
                               systems (different layout for ints and floats)
                               are currently not supported.
-NANOPB_INTERNALS               Set this to expose the field encoder functions
-                               that are hidden since nanopb-0.1.3. Starting
-                               with nanopb-0.2.4, this flag does nothing. Use
-                               the newer functions that have better interface.
+PB_ENABLE_MALLOC               Set this to enable dynamic allocation support
+                               in the decoder.
 PB_MAX_REQUIRED_FIELDS         Maximum number of required fields to check for
                               presence. Default value is 64. Increases stack
                               usage 1 byte per every 8 fields. Compiler
@@ -77,8 +75,9 @@ max_count                      Allocated number of entries in arrays
                               (*repeated* fields).
 type                           Type of the generated field. Default value
                               is *FT_DEFAULT*, which selects automatically.
-                               You can use *FT_CALLBACK*, *FT_STATIC* or
-                               *FT_IGNORE* to force a callback field, a static
+                               You can use *FT_CALLBACK*, *FT_POINTER*,
+                               *FT_STATIC* or *FT_IGNORE* to force a callback
+                               field, a dynamically allocated field, a static
                               field or to completely ignore the field.
 long_names                     Prefix the enum name to the enum value in
                               definitions, i.e. *EnumName_EnumValue*. Enabled
@@ -417,6 +416,17 @@ Encodes the contents of a structure as a protocol buffers message and writes it

 Normally pb_encode simply walks through the fields description array and serializes each field in turn. However, submessages must be serialized twice: first to calculate their size and then to actually write them to output. This causes some constraints for callback fields, which must return the same data on every call.

+pb_encode_delimited
+-------------------
+Calculates the length of the message, encodes it as varint and then encodes the message. ::
+
+    bool pb_encode_delimited(pb_ostream_t *stream, const pb_field_t fields[], const void *src_struct);
+
+(parameters are the same as for `pb_encode`_.)
+
+A common way to indicate the message length in Protocol Buffers is to prefix it with a varint.
+This function does this, and it is compatible with *parseDelimitedFrom* in Google's protobuf library.
+
 .. sidebar:: Encoding fields manually

    The functions with names *pb_encode_\** are used when dealing with callback fields. The typical reason for using callbacks is to have an array of unlimited size. In that case, `pb_encode`_ will call your callback function, which in turn will call *pb_encode_\** functions repeatedly to write out values.
@@ -579,6 +589,10 @@ In addition to EOF, the pb_decode implementation supports terminating a message

 For optional fields, this function applies the default value and sets *has_<field>* to false if the field is not present.

+If *PB_ENABLE_MALLOC* is defined, this function may allocate storage for any pointer type fields.
+In this case, you have to call `pb_release`_ to release the memory after you are done with the message.
+On error return `pb_decode` will release the memory itself.
+
 pb_decode_noinit
 ----------------
 Same as `pb_decode`_, except does not apply the default values to fields. ::
@@ -589,6 +603,35 @@ Same as `pb_decode`_, except does not apply the default values to fields. ::

 The destination structure should be filled with zeros before calling this function. Doing a *memset* manually can be slightly faster than using `pb_decode`_ if you don't need any default values.

+In addition to decoding a single message, this function can be used to merge two messages, so that
+values from previous message will remain if the new message does not contain a field.
+
+This function *will not* release the message even on error return. If you use *PB_ENABLE_MALLOC*,
+you will need to call `pb_release`_ yourself.
+
+pb_decode_delimited
+-------------------
+Same as `pb_decode`_, except that it first reads a varint with the length of the message. ::
+
+    bool pb_decode_delimited(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct);
+
+(parameters are the same as for `pb_decode`_.)
+
+A common method to indicate message size in Protocol Buffers is to prefix it with a varint.
+This function is compatible with *writeDelimitedTo* in the Google's Protocol Buffers library.
+
+pb_release
+----------
+Releases any dynamically allocated fields.
+
+    void pb_release(const pb_field_t fields[], void *dest_struct);
+
+:fields:        A field description array. Usually autogenerated.
+:dest_struct:   Pointer to structure where data will be stored.
+
+This function is only available if *PB_ENABLE_MALLOC* is defined. It will release any
+pointer type fields in the structure and set the pointers to NULL.
+
 pb_skip_varint
 --------------
 Skip a varint_ encoded integer without decoding it. ::
--- a/examples/network_server/client.c
+++ b/examples/network_server/client.c
@@ -23,9 +23,13 @@
 #include "fileproto.pb.h"
 #include "common.h"

+/* This callback function will be called once for each filename received
+ * from the server. The filenames will be printed out immediately, so that
+ * no memory has to be allocated for them.
+ */
 bool printfile_callback(pb_istream_t *stream, const pb_field_t *field, void **arg)
 {
-    FileInfo fileinfo;
+    FileInfo fileinfo = {};
    
    if (!pb_decode(stream, FileInfo_fields, &fileinfo))
        return false;
@@ -35,14 +39,20 @@ bool printfile_callback(pb_istream_t *stream, const pb_field_t *field, void **ar
    return true;
 }

+/* This function sends a request to socket 'fd' to list the files in
+ * directory given in 'path'. The results received from server will
+ * be printed to stdout.
+ */
 bool listdir(int fd, char *path)
 {
-    ListFilesRequest request;
-    ListFilesResponse response;
-    pb_istream_t input = pb_istream_from_socket(fd);
+    /* Construct and send the request to server */
+    {
+        ListFilesRequest request = {};
        pb_ostream_t output = pb_ostream_from_socket(fd);
        uint8_t zero = 0;
        
+        /* In our protocol, path is optional. If it is not given,
+         * the server will list the root directory. */
        if (path == NULL)
        {
            request.has_path = false;
@@ -59,15 +69,25 @@ bool listdir(int fd, char *path)
            strcpy(request.path, path);
        }
        
+        /* Encode the request. It is written to the socket immediately
+         * through our custom stream. */
        if (!pb_encode(&output, ListFilesRequest_fields, &request))
        {
-        fprintf(stderr, "Encoding failed.\n");
+            fprintf(stderr, "Encoding failed: %s\n", PB_GET_ERROR(&output));
            return false;
        }
        
        /* We signal the end of request with a 0 tag. */
        pb_write(&output, &zero, 1);
+    }
    
+    /* Read back the response from server */
+    {
+        ListFilesResponse response = {};
+        pb_istream_t input = pb_istream_from_socket(fd);
+        
+        /* Give a pointer to our callback function, which will handle the
+         * filenames as they arrive. */
        response.file.funcs.decode = &printfile_callback;
        
        if (!pb_decode(&input, ListFilesResponse_fields, &response))
@@ -76,11 +96,14 @@ bool listdir(int fd, char *path)
            return false;
        }
        
+        /* If the message from server decodes properly, but directory was
+         * not found on server side, we get path_error == true. */
        if (response.path_error)
        {
            fprintf(stderr, "Server reported error.\n");
            return false;
        }
+    }
    
    return true;
 }
@@ -96,6 +119,7 @@ int main(int argc, char **argv)
    
    sockfd = socket(AF_INET, SOCK_STREAM, 0);
    
+    /* Connect to server running on localhost:1234 */
    memset(&servaddr, 0, sizeof(servaddr));
    servaddr.sin_family = AF_INET;
    servaddr.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
@@ -107,9 +131,11 @@ int main(int argc, char **argv)
        return 1;
    }
    
+    /* Send the directory listing request */
    if (!listdir(sockfd, path))
        return 2;
    
+    /* Close connection */
    close(sockfd);
    
    return 0;
--- a/examples/network_server/server.c
+++ b/examples/network_server/server.c
@@ -23,11 +23,16 @@
 #include "fileproto.pb.h"
 #include "common.h"

+/* This callback function will be called once during the encoding.
+ * It will write out any number of FileInfo entries, without consuming unnecessary memory.
+ * This is accomplished by fetching the filenames one at a time and encoding them
+ * immediately.
+ */
 bool listdir_callback(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
 {
    DIR *dir = (DIR*) *arg;
    struct dirent *file;
-    FileInfo fileinfo;
+    FileInfo fileinfo = {};
    
    while ((file = readdir(dir)) != NULL)
    {
@@ -35,9 +40,12 @@ bool listdir_callback(pb_ostream_t *stream, const pb_field_t *field, void * cons
        strncpy(fileinfo.name, file->d_name, sizeof(fileinfo.name));
        fileinfo.name[sizeof(fileinfo.name) - 1] = '\0';
        
+        /* This encodes the header for the field, based on the constant info
+         * from pb_field_t. */
        if (!pb_encode_tag_for_field(stream, field))
            return false;
        
+        /* This encodes the data for the field, based on our FileInfo structure. */
        if (!pb_encode_submessage(stream, FileInfo_fields, &fileinfo))
            return false;
    }
@@ -45,13 +53,18 @@ bool listdir_callback(pb_ostream_t *stream, const pb_field_t *field, void * cons
    return true;
 }

+/* Handle one arriving client connection.
+ * Clients are expected to send a ListFilesRequest, terminated by a '0'.
+ * Server will respond with a ListFilesResponse message.
+ */
 void handle_connection(int connfd)
 {
-    ListFilesRequest request;
-    ListFilesResponse response;
+    DIR *directory = NULL;
+    
+    /* Decode the message from the client and open the requested directory. */
+    {
+        ListFilesRequest request = {};
        pb_istream_t input = pb_istream_from_socket(connfd);
-    pb_ostream_t output = pb_ostream_from_socket(connfd);
-    DIR *directory;
        
        if (!pb_decode(&input, ListFilesRequest_fields, &request))
        {
@@ -60,19 +73,26 @@ void handle_connection(int connfd)
        }
        
        directory = opendir(request.path);
-    
        printf("Listing directory: %s\n", request.path);
+    }
+    
+    /* List the files in the directory and transmit the response to client */
+    {
+        ListFilesResponse response = {};
+        pb_ostream_t output = pb_ostream_from_socket(connfd);
        
        if (directory == NULL)
        {
            perror("opendir");
            
+            /* Directory was not found, transmit error status */
            response.has_path_error = true;
            response.path_error = true;
            response.file.funcs.encode = NULL;
        }
        else
        {
+            /* Directory was found, transmit filenames */
            response.has_path_error = false;
            response.file.funcs.encode = &listdir_callback;
            response.file.arg = directory;
@@ -80,8 +100,12 @@ void handle_connection(int connfd)
        
        if (!pb_encode(&output, ListFilesResponse_fields, &response))
        {
-        printf("Encoding failed.\n");
+            printf("Encoding failed: %s\n", PB_GET_ERROR(&output));
        }
+    }
+    
+    if (directory != NULL)
+        closedir(directory);
 }

 int main(int argc, char **argv)
@@ -90,8 +114,8 @@ int main(int argc, char **argv)
    struct sockaddr_in servaddr;
    int reuse = 1;
    
+    /* Listen on localhost:1234 for TCP connections */
    listenfd = socket(AF_INET, SOCK_STREAM, 0);
-    
    setsockopt(listenfd, SOL_SOCKET, SO_REUSEADDR, &reuse, sizeof(reuse));
    
    memset(&servaddr, 0, sizeof(servaddr));
@@ -112,6 +136,7 @@ int main(int argc, char **argv)
    
    for(;;)
    {
+        /* Wait for a client */
        connfd = accept(listenfd, NULL, NULL);
        
        if (connfd < 0)
@@ -128,4 +153,6 @@ int main(int argc, char **argv)
        
        close(connfd);
    }
+    
+    return 0;
 }
--- a/extra/pb_syshdr.h
+++ b/extra/pb_syshdr.h
@@ -53,6 +53,16 @@ typedef int bool;

 #endif

+/* stdlib.h subset */
+#ifdef PB_ENABLE_MALLOC
+#ifdef HAVE_STDLIB_H
+#include <stdlib.h>
+#else
+void *realloc(void *ptr, size_t size);
+void free(void *ptr);
+#endif
+#endif
+
 /* string.h subset */
 #ifdef HAVE_STRING_H
 #include <string.h>
--- a/generator/nanopb_generator.py
+++ b/generator/nanopb_generator.py
@@ -1,7 +1,7 @@
 #!/usr/bin/python

 '''Generate header file for nanopb from a ProtoBuf FileDescriptorSet.'''
-nanopb_version = "nanopb-0.2.5"
+nanopb_version = "nanopb-0.2.9.2"

 import sys

@@ -15,6 +15,7 @@ except:

 try:
    import google.protobuf.text_format as text_format
+    import google.protobuf.descriptor_pb2 as descriptor
 except:
    sys.stderr.write('''
         *************************************************************
@@ -26,7 +27,7 @@ except:

 try:
    import proto.nanopb_pb2 as nanopb_pb2
-    import proto.descriptor_pb2 as descriptor
+    import proto.plugin_pb2 as plugin_pb2
 except:
    sys.stderr.write('''
         ********************************************************************
@@ -36,7 +37,6 @@ except:
    ''' + '\n')
    raise

-
 # ---------------------------------------------------------------------------
 #                     Generation of single fields
 # ---------------------------------------------------------------------------
@@ -169,6 +169,7 @@ class Field:
        self.max_count = None
        self.array_decl = ""
        self.enc_size = None
+        self.ctype = None
        
        # Parse field options
        if field_options.HasField("max_size"):
@@ -245,7 +246,7 @@ class Field:
                self.ctype = self.struct_name + self.name + 't'
                self.enc_size = varint_max_size(self.max_size) + self.max_size
            elif self.allocation == 'POINTER':
-                self.ctype = 'pb_bytes_ptr_t'
+                self.ctype = 'pb_bytes_array_t'
        elif desc.type == FieldD.TYPE_MESSAGE:
            self.pbtype = 'MESSAGE'
            self.ctype = self.submsgname = names_from_type_name(desc.type_name)
@@ -265,8 +266,8 @@ class Field:
            if self.pbtype == 'MESSAGE':
                # Use struct definition, so recursive submessages are possible
                result += '    struct _%s *%s;' % (self.ctype, self.name)
-            elif self.rules == 'REPEATED' and self.pbtype == 'STRING':
-                # String arrays need to be defined as pointers to pointers
+            elif self.rules == 'REPEATED' and self.pbtype in ['STRING', 'BYTES']:
+                # String/bytes arrays need to be defined as pointers to pointers
                result += '    %s **%s;' % (self.ctype, self.name)
            else:
                result += '    %s *%s;' % (self.ctype, self.name)
@@ -291,30 +292,64 @@ class Field:
            result = None
        return result
    
+    def get_initializer(self, null_init):
+        '''Return literal expression for this field's default value.'''
+        
+        if self.pbtype == 'MESSAGE':
+            if null_init:
+                return '%s_init_zero' % self.ctype
+            else:
+                return '%s_init_default' % self.ctype
+        
+        if self.default is None or null_init:
+            if self.pbtype == 'STRING':
+                return '""'
+            elif self.pbtype == 'BYTES':
+                return '{0, {0}}'
+            elif self.pbtype == 'ENUM':
+                return '(%s)0' % self.ctype
+            else:
+                return '0'
+        
+        default = str(self.default)
+        
+        if self.pbtype == 'STRING':
+            default = default.encode('utf-8').encode('string_escape')
+            default = default.replace('"', '\\"')
+            default = '"' + default + '"'
+        elif self.pbtype == 'BYTES':
+            data = default.decode('string_escape')
+            data = ['0x%02x' % ord(c) for c in data]
+            if len(data) == 0:
+                default = '{0, {0}}'
+            else:
+                default = '{%d, {%s}}' % (len(data), ','.join(data))
+        elif self.pbtype in ['FIXED32', 'UINT32']:
+            default += 'u'
+        elif self.pbtype in ['FIXED64', 'UINT64']:
+            default += 'ull'
+        elif self.pbtype in ['SFIXED64', 'INT64']:
+            default += 'll'
+        
+        return default
+    
    def default_decl(self, declaration_only = False):
        '''Return definition for this field's default value.'''
        if self.default is None:
            return None

-        ctype, default = self.ctype, self.default
+        ctype = self.ctype
+        default = self.get_initializer(False)
        array_decl = ''
        
        if self.pbtype == 'STRING':
            if self.allocation != 'STATIC':
                return None # Not implemented
-        
            array_decl = '[%d]' % self.max_size
-            default = str(self.default).encode('string_escape')
-            default = default.replace('"', '\\"')
-            default = '"' + default + '"'
        elif self.pbtype == 'BYTES':
            if self.allocation != 'STATIC':
                return None # Not implemented
        
-            data = self.default.decode('string_escape')
-            data = ['0x%02x' % ord(c) for c in data]
-            default = '{%d, {%s}}' % (len(data), ','.join(data))
-        
        if declaration_only:
            return 'extern const %s %s_default%s;' % (ctype, self.struct_name + self.name, array_decl)
        else:
@@ -344,6 +379,8 @@ class Field:
            result += '0)'
        elif self.pbtype in ['BYTES', 'STRING'] and self.allocation != 'STATIC':
            result += '0)' # Arbitrary size default values not implemented
+        elif self.rules == 'OPTEXT':
+            result += '0)' # Default value for extensions is not implemented
        else:
            result += '&%s_default)' % (self.struct_name + self.name)
        
@@ -544,6 +581,34 @@ class Message:
                result += types + '\n'
        return result
    
+    def get_initializer(self, null_init):
+        if not self.ordered_fields:
+            return '{0}'
+    
+        parts = []
+        for field in self.ordered_fields:
+            if field.allocation == 'STATIC':
+                if field.rules == 'REPEATED':
+                    parts.append('0')
+                    parts.append('{'
+                                 + ', '.join([field.get_initializer(null_init)] * field.max_count)
+                                 + '}')
+                elif field.rules == 'OPTIONAL':
+                    parts.append('false')
+                    parts.append(field.get_initializer(null_init))
+                else:
+                    parts.append(field.get_initializer(null_init))
+            elif field.allocation == 'POINTER':
+                if field.rules == 'REPEATED':
+                    parts.append('0')
+                parts.append('NULL')
+            elif field.allocation == 'CALLBACK':
+                if field.pbtype == 'EXTENSION':
+                    parts.append('NULL')
+                else:
+                    parts.append('{{NULL}, NULL}')
+        return '{' + ', '.join(parts) + '}'
+    
    def default_decl(self, declaration_only = False):
        result = ""
        for field in self.fields:
@@ -630,13 +695,17 @@ def parse_file(fdesc, file_options):
    
    for names, message in iterate_messages(fdesc, base_name):
        message_options = get_nanopb_suboptions(message, file_options, names)
+        
+        if message_options.skip_message:
+            continue
+        
        messages.append(Message(names, message, message_options))
        for enum in message.enum_type:
            enum_options = get_nanopb_suboptions(enum, message_options, names + enum.name)
            enums.append(Enum(names, enum, enum_options))
    
    for names, extension in iterate_extensions(fdesc, base_name):
-        field_options = get_nanopb_suboptions(extension, file_options, names)
+        field_options = get_nanopb_suboptions(extension, file_options, names + extension.name)
        if field_options.type != nanopb_pb2.FT_IGNORE:
            extensions.append(ExtensionField(names, extension, field_options))
    
@@ -698,6 +767,9 @@ def generate_header(dependencies, headername, enums, messages, extensions, optio
    '''
    
    yield '/* Automatically generated nanopb header */\n'
+    if options.notimestamp:
+        yield '/* Generated by %s */\n\n' % (nanopb_version)
+    else:
        yield '/* Generated by %s at %s. */\n\n' % (nanopb_version, time.asctime())
    
    symbol = make_identifier(headername)
@@ -712,7 +784,7 @@ def generate_header(dependencies, headername, enums, messages, extensions, optio
    
    for dependency in dependencies:
        noext = os.path.splitext(dependency)[0]
-        yield options.genformat % (noext + '.' + options.extension + '.h')
+        yield options.genformat % (noext + options.extension + '.h')
        yield '\n'

    yield '#ifdef __cplusplus\n'
@@ -739,6 +811,15 @@ def generate_header(dependencies, headername, enums, messages, extensions, optio
        yield msg.default_decl(True)
    yield '\n'
    
+    yield '/* Initializer values for message structs */\n'
+    for msg in messages:
+        identifier = '%s_init_default' % msg.name
+        yield '#define %-40s %s\n' % (identifier, msg.get_initializer(False))
+    for msg in messages:
+        identifier = '%s_init_zero' % msg.name
+        yield '#define %-40s %s\n' % (identifier, msg.get_initializer(True))
+    yield '\n'
+    
    yield '/* Field tags (for use in manual encoding/decoding) */\n'
    for msg in sort_dependencies(messages):
        for field in msg.fields:
@@ -771,6 +852,9 @@ def generate_source(headername, enums, messages, extensions, options):
    '''Generate content for a source file.'''
    
    yield '/* Automatically generated nanopb constant definitions */\n'
+    if options.notimestamp:
+        yield '/* Generated by %s */\n\n' % (nanopb_version)
+    else:
        yield '/* Generated by %s at %s. */\n\n' % (nanopb_version, time.asctime())
    yield options.genformat % (headername)
    yield '\n'
@@ -815,16 +899,6 @@ def generate_source(headername, enums, messages, extensions, options):
    if worst > 255 or checks:
        yield '\n/* Check that field information fits in pb_field_t */\n'
        
-        if worst < 65536:
-            yield '#if !defined(PB_FIELD_16BIT) && !defined(PB_FIELD_32BIT)\n'
-            if worst > 255:
-                yield '#error Field descriptor for %s is too large. Define PB_FIELD_16BIT to fix this.\n' % worst_field
-            else:
-                assertion = ' && '.join(str(c) + ' < 256' for c in checks)
-                msgs = '_'.join(str(n) for n in checks_msgnames)
-                yield 'STATIC_ASSERT((%s), YOU_MUST_DEFINE_PB_FIELD_16BIT_FOR_MESSAGES_%s)\n'%(assertion,msgs)
-            yield '#endif\n\n'
-        
        if worst > 65535 or checks:
            yield '#if !defined(PB_FIELD_32BIT)\n'
            if worst > 65535:
@@ -832,8 +906,32 @@ def generate_source(headername, enums, messages, extensions, options):
            else:
                assertion = ' && '.join(str(c) + ' < 65536' for c in checks)
                msgs = '_'.join(str(n) for n in checks_msgnames)
+                yield '/* If you get an error here, it means that you need to define PB_FIELD_32BIT\n'
+                yield ' * compile-time option. You can do that in pb.h or on compiler command line.\n'
+                yield ' * \n'
+                yield ' * The reason you need to do this is that some of your messages contain tag\n'
+                yield ' * numbers or field sizes that are larger than what can fit in 8 or 16 bit\n'
+                yield ' * field descriptors.\n'
+                yield ' */\n'
                yield 'STATIC_ASSERT((%s), YOU_MUST_DEFINE_PB_FIELD_32BIT_FOR_MESSAGES_%s)\n'%(assertion,msgs)
-            yield '#endif\n'
+            yield '#endif\n\n'
+        
+        if worst < 65536:
+            yield '#if !defined(PB_FIELD_16BIT) && !defined(PB_FIELD_32BIT)\n'
+            if worst > 255:
+                yield '#error Field descriptor for %s is too large. Define PB_FIELD_16BIT to fix this.\n' % worst_field
+            else:
+                assertion = ' && '.join(str(c) + ' < 256' for c in checks)
+                msgs = '_'.join(str(n) for n in checks_msgnames)
+                yield '/* If you get an error here, it means that you need to define PB_FIELD_16BIT\n'
+                yield ' * compile-time option. You can do that in pb.h or on compiler command line.\n'
+                yield ' * \n'
+                yield ' * The reason you need to do this is that some of your messages contain tag\n'
+                yield ' * numbers or field sizes that are larger than what can fit in the default\n'
+                yield ' * 8 bit descriptors.\n'
+                yield ' */\n'
+                yield 'STATIC_ASSERT((%s), YOU_MUST_DEFINE_PB_FIELD_16BIT_FOR_MESSAGES_%s)\n'%(assertion,msgs)
+            yield '#endif\n\n'
    
    # Add check for sizeof(double)
    has_double = False
@@ -879,6 +977,7 @@ class Globals:
    '''Ugly global variables, should find a good way to pass these.'''
    verbose_options = False
    separate_options = []
+    matched_namemasks = set()

 def get_nanopb_suboptions(subdesc, options, name):
    '''Get copy of options, and merge information from subdesc.'''
@@ -889,6 +988,7 @@ def get_nanopb_suboptions(subdesc, options, name):
    dotname = '.'.join(name.parts)
    for namemask, options in Globals.separate_options:
        if fnmatch(dotname, namemask):
+            Globals.matched_namemasks.add(namemask)
            new_options.MergeFrom(options)
    
    # Handle options defined in .proto
@@ -908,8 +1008,8 @@ def get_nanopb_suboptions(subdesc, options, name):
        new_options.MergeFrom(ext)
    
    if Globals.verbose_options:
-        print "Options for " + dotname + ":"
-        print text_format.MessageToString(new_options)
+        sys.stderr.write("Options for " + dotname + ": ")
+        sys.stderr.write(text_format.MessageToString(new_options) + "\n")
    
    return new_options

@@ -928,8 +1028,8 @@ optparser = OptionParser(
             "Output will be written to file.pb.h and file.pb.c.")
 optparser.add_option("-x", dest="exclude", metavar="FILE", action="append", default=[],
    help="Exclude file from generated #include list.")
-optparser.add_option("-e", "--extension", dest="extension", metavar="EXTENSION", default="pb",
-    help="Set extension to use instead of 'pb' for generated files. [default: %default]")
+optparser.add_option("-e", "--extension", dest="extension", metavar="EXTENSION", default=".pb",
+    help="Set extension to use instead of '.pb' for generated files. [default: %default]")
 optparser.add_option("-f", "--options-file", dest="options_file", metavar="FILE", default="%s.options",
    help="Set name of a separate generator options file.")
 optparser.add_option("-Q", "--generated-include-format", dest="genformat",
@@ -938,6 +1038,8 @@ optparser.add_option("-Q", "--generated-include-format", dest="genformat",
 optparser.add_option("-L", "--library-include-format", dest="libformat",
    metavar="FORMAT", default='#include <%s>\n',
    help="Set format string to use for including the nanopb pb.h header. [default: %default]")
+optparser.add_option("-T", "--no-timestamp", dest="notimestamp", action="store_true", default=False,
+    help="Don't add timestamp to .pb.h and .pb.c preambles")
 optparser.add_option("-q", "--quiet", dest="quiet", action="store_true", default=False,
    help="Don't print anything except errors.")
 optparser.add_option("-v", "--verbose", dest="verbose", action="store_true", default=False,
@@ -967,28 +1069,38 @@ def process_file(filename, fdesc, options):
        fdesc = descriptor.FileDescriptorSet.FromString(data).file[0]
    
    # Check if there is a separate .options file
+    had_abspath = False
    try:
        optfilename = options.options_file % os.path.splitext(filename)[0]
    except TypeError:
        # No %s specified, use the filename as-is
        optfilename = options.options_file
-    
-    if options.verbose:
-        print 'Reading options from ' + optfilename
+        had_abspath = True

    if os.path.isfile(optfilename):
+        if options.verbose:
+            sys.stderr.write('Reading options from ' + optfilename + '\n')
+
        Globals.separate_options = read_options_file(open(optfilename, "rU"))
    else:
+        # If we are given a full filename and it does not exist, give an error.
+        # However, don't give error when we automatically look for .options file
+        # with the same name as .proto.
+        if options.verbose or had_abspath:
+            sys.stderr.write('Options file not found: ' + optfilename)
+
        Globals.separate_options = []

+    Globals.matched_namemasks = set()
+    
    # Parse the file
    file_options = get_nanopb_suboptions(fdesc, toplevel_options, Names([filename]))
    enums, messages, extensions = parse_file(fdesc, file_options)

    # Decide the file names
    noext = os.path.splitext(filename)[0]
-    headername = noext + '.' + options.extension + '.h'
-    sourcename = noext + '.' + options.extension + '.c'
+    headername = noext + options.extension + '.h'
+    sourcename = noext + options.extension + '.c'
    headerbasename = os.path.basename(headername)
    
    # List of .proto files that should not be included in the C header file
@@ -1002,6 +1114,14 @@ def process_file(filename, fdesc, options):
    sourcedata = ''.join(generate_source(headerbasename, enums,
                                         messages, extensions, options))

+    # Check if there were any lines in .options that did not match a member
+    unmatched = [n for n,o in Globals.separate_options if n not in Globals.matched_namemasks]
+    if unmatched and not options.quiet:
+        sys.stderr.write("Following patterns in " + optfilename + " did not match any fields: "
+                         + ', '.join(unmatched) + "\n")
+        if not Globals.verbose_options:
+            sys.stderr.write("Use  protoc --nanopb-out=-v:.   to see a list of the field names.\n")
+
    return {'headername': headername, 'headerdata': headerdata,
            'sourcename': sourcename, 'sourcedata': sourcedata}
    
@@ -1023,7 +1143,8 @@ def main_cli():
        results = process_file(filename, None, options)
        
        if not options.quiet:
-            print "Writing to " + results['headername'] + " and " + results['sourcename']
+            sys.stderr.write("Writing to " + results['headername'] + " and "
+                             + results['sourcename'] + "\n")
    
        open(results['headername'], 'w').write(results['headerdata'])
        open(results['sourcename'], 'w').write(results['sourcedata'])        
@@ -1038,7 +1159,6 @@ def main_plugin():
        msvcrt.setmode(sys.stdin.fileno(), os.O_BINARY)
        msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)
    
-    import proto.plugin_pb2 as plugin_pb2
    data = sys.stdin.read()
    request = plugin_pb2.CodeGeneratorRequest.FromString(data)
    
@@ -1046,10 +1166,7 @@ def main_plugin():
    args = shlex.split(request.parameter)
    options, dummy = optparser.parse_args(args)
    
-    # We can't go printing stuff to stdout
-    Globals.verbose_options = False
-    options.verbose = False
-    options.quiet = True
+    Globals.verbose_options = options.verbose
    
    response = plugin_pb2.CodeGeneratorResponse()
    
--- a/generator/proto/Makefile
+++ b/generator/proto/Makefile
@@ -1,4 +1,4 @@
-all: nanopb_pb2.py plugin_pb2.py descriptor_pb2.py
+all: nanopb_pb2.py plugin_pb2.py

 %_pb2.py: %.proto
 	protoc --python_out=. $<
--- a/generator/proto/google/protobuf/descriptor.proto
+++ b/generator/proto/google/protobuf/descriptor.proto
--- a/generator/proto/nanopb.proto
+++ b/generator/proto/nanopb.proto
@@ -5,7 +5,7 @@
 // These are used by nanopb to generate statically allocable structures
 // for memory-limited environments.

-import "descriptor.proto";
+import "google/protobuf/descriptor.proto";

 option java_package = "fi.kapsi.koti.jpa.nanopb";

@@ -37,6 +37,9 @@ message NanoPBOptions {
  // Note: this cannot be used on CPUs that break on unaligned
  // accesses to variables.
  optional bool packed_struct = 5 [default = false];
+  
+  // Skip this message
+  optional bool skip_message = 6 [default = false];
 }

 // Extensions to protoc 'Descriptor' type in order to define options
--- a/generator/proto/plugin.proto
+++ b/generator/proto/plugin.proto
@@ -46,7 +46,7 @@

 package google.protobuf.compiler;

-import "descriptor.proto";
+import "google/protobuf/descriptor.proto";

 // An encoded CodeGeneratorRequest is written to the plugin's stdin.
 message CodeGeneratorRequest {
--- a/pb.h
+++ b/pb.h
@@ -10,6 +10,9 @@
 * uncommenting the lines, or on the compiler command line.      *
 *****************************************************************/

+/* Enable support for dynamically allocated fields */
+/* #define PB_ENABLE_MALLOC 1 */
+
 /* Define this if your CPU architecture is big endian, i.e. it
 * stores the most-significant byte first. */
 /* #define __BIG_ENDIAN__ 1 */
@@ -43,7 +46,7 @@

 /* Version of the nanopb library. Just in case you want to check it in
 * your own program. */
-#define NANOPB_VERSION nanopb-0.2.5
+#define NANOPB_VERSION nanopb-0.2.9.2

 /* Include all the system headers needed by nanopb. You will need the
 * definitions of the following:
@@ -63,6 +66,10 @@
 #include <stddef.h>
 #include <stdbool.h>
 #include <string.h>
+
+#ifdef PB_ENABLE_MALLOC
+#include <stdlib.h>
+#endif
 #endif

 /* Macro for defining packed structures (compiler dependent).
@@ -73,8 +80,8 @@
 #   define PB_PACKED_STRUCT_START
 #   define PB_PACKED_STRUCT_END
 #   define pb_packed __attribute__((packed))
-#elif defined(__ICCARM__)
-    /* For IAR ARM compiler */
+#elif defined(__ICCARM__) || defined(__CC_ARM)
+    /* For IAR ARM and Keil MDK-ARM compilers */
 #   define PB_PACKED_STRUCT_START _Pragma("pack(push, 1)")
 #   define PB_PACKED_STRUCT_END _Pragma("pack(pop)")
 #   define pb_packed
@@ -96,8 +103,14 @@
 #endif

 /* Compile-time assertion, used for checking compatible compilation options.
- * If this fails on your compiler for some reason, use #define STATIC_ASSERT
- * to disable it. */
+ * If this does not work properly on your compiler, use #define STATIC_ASSERT
+ * to disable it.
+ *
+ * But before doing that, check carefully the error message / place where it
+ * comes from to see if the error has a real cause. Unfortunately the error
+ * message is not always very clear to read, but you can see the reason better
+ * in the place where the STATIC_ASSERT macro was called.
+ */
 #ifndef STATIC_ASSERT
 #define STATIC_ASSERT(COND,MSG) typedef char STATIC_ASSERT_MSG(MSG, __LINE__, __COUNTER__)[(COND)?1:-1];
 #define STATIC_ASSERT_MSG(MSG, LINE, COUNTER) STATIC_ASSERT_MSG_(MSG, LINE, COUNTER)
@@ -210,7 +223,11 @@ struct _pb_field_t {
 PB_PACKED_STRUCT_END

 /* Make sure that the standard integer types are of the expected sizes.
- * All kinds of things may break otherwise.. atleast all fixed* types. */
+ * All kinds of things may break otherwise.. atleast all fixed* types.
+ *
+ * If you get errors here, it probably means that your stdint.h is not
+ * correct for your platform.
+ */
 STATIC_ASSERT(sizeof(int8_t) == 1, INT8_T_WRONG_SIZE)
 STATIC_ASSERT(sizeof(uint8_t) == 1, UINT8_T_WRONG_SIZE)
 STATIC_ASSERT(sizeof(int16_t) == 2, INT16_T_WRONG_SIZE)
@@ -224,21 +241,15 @@ STATIC_ASSERT(sizeof(uint64_t) == 8, UINT64_T_WRONG_SIZE)
 * It has the number of bytes in the beginning, and after that an array.
 * Note that actual structs used will have a different length of bytes array.
 */
+#define PB_BYTES_ARRAY_T(n) struct { size_t size; uint8_t bytes[n]; }
+#define PB_BYTES_ARRAY_T_ALLOCSIZE(n) ((size_t)n + offsetof(pb_bytes_array_t, bytes))
+
 struct _pb_bytes_array_t {
    size_t size;
    uint8_t bytes[1];
 };
 typedef struct _pb_bytes_array_t pb_bytes_array_t;

-/* Same, except for pointer-type fields. There is no need to variable struct
- * length in this case.
- */
-struct _pb_bytes_ptr_t {
-    size_t size;
-    uint8_t *bytes;
-};
-typedef struct _pb_bytes_ptr_t pb_bytes_ptr_t;
-
 /* This structure is used for giving the callback function.
 * It is stored in the message structure and filled in by the method that
 * calls pb_decode.
@@ -330,8 +341,23 @@ struct _pb_extension_t {
     * If this extension does not match a field, the next handler is
     * automatically called. */
    pb_extension_t *next;
+
+    /* The decoder sets this to true if the extension was found.
+     * Ignored for encoding. */
+    bool found;
 };

+/* Memory allocation functions to use. You can define pb_realloc and
+ * pb_free to custom functions if you want. */
+#ifdef PB_ENABLE_MALLOC
+#   ifndef pb_realloc
+#       define pb_realloc(ptr, size) realloc(ptr, size)
+#   endif
+#   ifndef pb_free
+#       define pb_free(ptr) free(ptr)
+#   endif
+#endif
+
 /* These macros are used to declare pb_field_t's in the constant array. */
 /* Size of a structure member, in bytes. */
 #define pb_membersize(st, m) (sizeof ((st*)0)->m)
@@ -474,7 +500,11 @@ struct _pb_extension_t {
 * messages if not used.
 */
 #ifdef PB_NO_ERRMSG
-#define PB_RETURN_ERROR(stream,msg) return false
+#define PB_RETURN_ERROR(stream,msg) \
+    do {\
+        UNUSED(stream); \
+        return false; \
+    } while(0)
 #define PB_GET_ERROR(stream) "(errmsg disabled)"
 #else
 #define PB_RETURN_ERROR(stream,msg) \
--- a/pb_decode.c
+++ b/pb_decode.c
@@ -13,7 +13,6 @@
    #define checkreturn __attribute__((warn_unused_result))
 #endif

-#define NANOPB_INTERNALS
 #include "pb.h"
 #include "pb_decode.h"

@@ -43,6 +42,7 @@ static bool checkreturn pb_field_find(pb_field_iterator_t *iter, uint32_t tag);
 static bool checkreturn decode_static_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iterator_t *iter);
 static bool checkreturn decode_callback_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iterator_t *iter);
 static bool checkreturn decode_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iterator_t *iter);
+static void iter_from_extension(pb_field_iterator_t *iter, pb_extension_t *extension);
 static bool checkreturn default_extension_decoder(pb_istream_t *stream, pb_extension_t *extension, uint32_t tag, pb_wire_type_t wire_type);
 static bool checkreturn decode_extension(pb_istream_t *stream, uint32_t tag, pb_wire_type_t wire_type, pb_field_iterator_t *iter);
 static bool checkreturn find_extension_field(pb_field_iterator_t *iter);
@@ -58,6 +58,11 @@ static bool checkreturn pb_dec_submessage(pb_istream_t *stream, const pb_field_t
 static bool checkreturn pb_skip_varint(pb_istream_t *stream);
 static bool checkreturn pb_skip_string(pb_istream_t *stream);

+#ifdef PB_ENABLE_MALLOC
+static bool checkreturn allocate_field(pb_istream_t *stream, void *pData, size_t data_size, size_t array_size);
+static void pb_release_single_field(const pb_field_iterator_t *iter);
+#endif
+
 /* --- Function pointers to field decoders ---
 * Order in the array must match pb_action_t LTYPE numbering.
 */
@@ -130,7 +135,7 @@ bool checkreturn pb_read(pb_istream_t *stream, uint8_t *buf, size_t count)
 * This is an optimization for the varint decoding. */
 static bool checkreturn pb_readbyte(pb_istream_t *stream, uint8_t *buf)
 {
-    if (!stream->bytes_left)
+    if (stream->bytes_left == 0)
        PB_RETURN_ERROR(stream, "end-of-stream");

 #ifndef PB_BUFFER_ONLY
@@ -174,7 +179,7 @@ static bool checkreturn pb_decode_varint32(pb_istream_t *stream, uint32_t *dest)
    if (!pb_readbyte(stream, &byte))
        return false;
    
-    if (!(byte & 0x80))
+    if ((byte & 0x80) == 0)
    {
        /* Quick case, 1 byte value */
        result = byte;
@@ -359,6 +364,10 @@ static bool pb_field_next(pb_field_iterator_t *iter)
    {
        prev_size *= iter->pos->array_size;
    }
+    else if (PB_ATYPE(iter->pos->type) == PB_ATYPE_POINTER)
+    {
+        prev_size = sizeof(void*);
+    }
    
    if (iter->pos->tag == 0)
        return false; /* Only happens with empty message types */
@@ -393,7 +402,7 @@ static bool checkreturn pb_field_find(pb_field_iterator_t *iter, uint32_t tag)
        {
            return true;
        }
-        pb_field_next(iter);
+        (void)pb_field_next(iter);
    } while (iter->field_index != start);
    
    return false;
@@ -431,7 +440,7 @@ static bool checkreturn decode_static_field(pb_istream_t *stream, pb_wire_type_t
                if (!pb_make_string_substream(stream, &substream))
                    return false;
                
-                while (substream.bytes_left && *size < iter->pos->array_size)
+                while (substream.bytes_left > 0 && *size < iter->pos->array_size)
                {
                    void *pItem = (uint8_t*)iter->pData + iter->pos->data_size * (*size);
                    if (!func(&substream, iter->pos, pItem))
@@ -465,6 +474,164 @@ static bool checkreturn decode_static_field(pb_istream_t *stream, pb_wire_type_t
    }
 }

+#ifdef PB_ENABLE_MALLOC
+/* Allocate storage for the field and store the pointer at iter->pData.
+ * array_size is the number of entries to reserve in an array.
+ * Zero size is not allowed, use pb_free() for releasing.
+ */
+static bool checkreturn allocate_field(pb_istream_t *stream, void *pData, size_t data_size, size_t array_size)
+{    
+    void *ptr = *(void**)pData;
+    
+    if (data_size == 0 || array_size == 0)
+        PB_RETURN_ERROR(stream, "invalid size");
+    
+    /* Check for multiplication overflows.
+     * This code avoids the costly division if the sizes are small enough.
+     * Multiplication is safe as long as only half of bits are set
+     * in either multiplicand.
+     */
+    {
+        const size_t check_limit = (size_t)1 << (sizeof(size_t) * 4);
+        if (data_size >= check_limit || array_size >= check_limit)
+        {
+            const size_t size_max = (size_t)-1;
+            if (size_max / array_size < data_size)
+            {
+                PB_RETURN_ERROR(stream, "size too large");
+            }
+        }
+    }
+    
+    /* Allocate new or expand previous allocation */
+    /* Note: on failure the old pointer will remain in the structure,
+     * the message must be freed by caller also on error return. */
+    ptr = pb_realloc(ptr, array_size * data_size);
+    if (ptr == NULL)
+        PB_RETURN_ERROR(stream, "realloc failed");
+    
+    *(void**)pData = ptr;
+    return true;
+}
+
+/* Clear a newly allocated item in case it contains a pointer, or is a submessage. */
+static void initialize_pointer_field(void *pItem, pb_field_iterator_t *iter)
+{
+    if (PB_LTYPE(iter->pos->type) == PB_LTYPE_STRING ||
+        PB_LTYPE(iter->pos->type) == PB_LTYPE_BYTES)
+    {
+        *(void**)pItem = NULL;
+    }
+    else if (PB_LTYPE(iter->pos->type) == PB_LTYPE_SUBMESSAGE)
+    {
+        pb_message_set_to_defaults((const pb_field_t *) iter->pos->ptr, pItem);
+    }
+}
+#endif
+
+static bool checkreturn decode_pointer_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iterator_t *iter)
+{
+#ifndef PB_ENABLE_MALLOC
+    UNUSED(wire_type);
+    UNUSED(iter);
+    PB_RETURN_ERROR(stream, "no malloc support");
+#else
+    pb_type_t type;
+    pb_decoder_t func;
+    
+    type = iter->pos->type;
+    func = PB_DECODERS[PB_LTYPE(type)];
+    
+    switch (PB_HTYPE(type))
+    {
+        case PB_HTYPE_REQUIRED:
+        case PB_HTYPE_OPTIONAL:
+            if (PB_LTYPE(type) == PB_LTYPE_SUBMESSAGE &&
+                *(void**)iter->pData != NULL)
+            {
+                /* Duplicate field, have to release the old allocation first. */
+                pb_release_single_field(iter);
+            }
+        
+            if (PB_LTYPE(type) == PB_LTYPE_STRING ||
+                PB_LTYPE(type) == PB_LTYPE_BYTES)
+            {
+                return func(stream, iter->pos, iter->pData);
+            }
+            else
+            {
+                if (!allocate_field(stream, iter->pData, iter->pos->data_size, 1))
+                    return false;
+                
+                initialize_pointer_field(*(void**)iter->pData, iter);
+                return func(stream, iter->pos, *(void**)iter->pData);
+            }
+    
+        case PB_HTYPE_REPEATED:
+            if (wire_type == PB_WT_STRING
+                && PB_LTYPE(type) <= PB_LTYPE_LAST_PACKABLE)
+            {
+                /* Packed array, multiple items come in at once. */
+                bool status = true;
+                size_t *size = (size_t*)iter->pSize;
+                size_t allocated_size = *size;
+                void *pItem;
+                pb_istream_t substream;
+                
+                if (!pb_make_string_substream(stream, &substream))
+                    return false;
+                
+                while (substream.bytes_left)
+                {
+                    if (*size + 1 > allocated_size)
+                    {
+                        /* Allocate more storage. This tries to guess the
+                         * number of remaining entries. Round the division
+                         * upwards. */
+                        allocated_size += (substream.bytes_left - 1) / iter->pos->data_size + 1;
+                        
+                        if (!allocate_field(&substream, iter->pData, iter->pos->data_size, allocated_size))
+                        {
+                            status = false;
+                            break;
+                        }
+                    }
+
+                    /* Decode the array entry */
+                    pItem = *(uint8_t**)iter->pData + iter->pos->data_size * (*size);
+                    initialize_pointer_field(pItem, iter);
+                    if (!func(&substream, iter->pos, pItem))
+                    {
+                        status = false;
+                        break;
+                    }
+                    (*size)++;
+                }
+                pb_close_string_substream(stream, &substream);
+                
+                return status;
+            }
+            else
+            {
+                /* Normal repeated field, i.e. only one item at a time. */
+                size_t *size = (size_t*)iter->pSize;
+                void *pItem;
+                
+                (*size)++;
+                if (!allocate_field(stream, iter->pData, iter->pos->data_size, *size))
+                    return false;
+            
+                pItem = *(uint8_t**)iter->pData + iter->pos->data_size * (*size - 1);
+                initialize_pointer_field(pItem, iter);
+                return func(stream, iter->pos, pItem);
+            }
+            
+        default:
+            PB_RETURN_ERROR(stream, "invalid field type");
+    }
+#endif
+}
+
 static bool checkreturn decode_callback_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iterator_t *iter)
 {
    pb_callback_t *pCallback = (pb_callback_t*)iter->pData;
@@ -519,6 +686,9 @@ static bool checkreturn decode_field(pb_istream_t *stream, pb_wire_type_t wire_t
        case PB_ATYPE_STATIC:
            return decode_static_field(stream, wire_type, iter);
        
+        case PB_ATYPE_POINTER:
+            return decode_pointer_field(stream, wire_type, iter);
+        
        case PB_ATYPE_CALLBACK:
            return decode_callback_field(stream, wire_type, iter);
        
@@ -527,6 +697,19 @@ static bool checkreturn decode_field(pb_istream_t *stream, pb_wire_type_t wire_t
    }
 }

+static void iter_from_extension(pb_field_iterator_t *iter, pb_extension_t *extension)
+{
+    const pb_field_t *field = (const pb_field_t*)extension->type->arg;
+    
+    iter->start = field;
+    iter->pos = field;
+    iter->field_index = 0;
+    iter->required_field_index = 0;
+    iter->dest_struct = extension->dest;
+    iter->pData = extension->dest;
+    iter->pSize = &extension->found;
+}
+
 /* Default handler for extension fields. Expects a pb_field_t structure
 * in extension->type->arg. */
 static bool checkreturn default_extension_decoder(pb_istream_t *stream,
@@ -534,19 +717,11 @@ static bool checkreturn default_extension_decoder(pb_istream_t *stream,
 {
    const pb_field_t *field = (const pb_field_t*)extension->type->arg;
    pb_field_iterator_t iter;
-    bool dummy;
    
    if (field->tag != tag)
        return true;
    
-    iter.start = field;
-    iter.pos = field;
-    iter.field_index = 0;
-    iter.required_field_index = 0;
-    iter.dest_struct = extension->dest;
-    iter.pData = extension->dest;
-    iter.pSize = &dummy;
-    
+    iter_from_extension(&iter, extension);
    return decode_field(stream, wire_type, &iter);
 }

@@ -558,7 +733,7 @@ static bool checkreturn decode_extension(pb_istream_t *stream,
    pb_extension_t *extension = *(pb_extension_t* const *)iter->pData;
    size_t pos = stream->bytes_left;
    
-    while (extension && pos == stream->bytes_left)
+    while (extension != NULL && pos == stream->bytes_left)
    {
        bool status;
        if (extension->type->decode)
@@ -585,7 +760,7 @@ static bool checkreturn find_extension_field(pb_field_iterator_t *iter)
    do {
        if (PB_LTYPE(iter->pos->type) == PB_LTYPE_EXTENSION)
            return true;
-        pb_field_next(iter);
+        (void)pb_field_next(iter);
    } while (iter->field_index != start);
    
    return false;
@@ -597,45 +772,60 @@ static void pb_message_set_to_defaults(const pb_field_t fields[], void *dest_str
    pb_field_iterator_t iter;
    pb_field_init(&iter, fields, dest_struct);
    
-    /* Initialize size/has fields and apply default values */
    do
    {
        pb_type_t type;
        type = iter.pos->type;
    
+        /* Avoid crash on empty message types (zero fields) */
        if (iter.pos->tag == 0)
            continue;
        
        if (PB_ATYPE(type) == PB_ATYPE_STATIC)
        {
-            /* Initialize the size field for optional/repeated fields to 0. */
            if (PB_HTYPE(type) == PB_HTYPE_OPTIONAL)
            {
+                /* Set has_field to false. Still initialize the optional field
+                 * itself also. */
                *(bool*)iter.pSize = false;
            }
            else if (PB_HTYPE(type) == PB_HTYPE_REPEATED)
            {
+                /* Set array count to 0, no need to initialize contents. */
                *(size_t*)iter.pSize = 0;
-                continue; /* Array is empty, no need to initialize contents */
+                continue;
            }
            
-            /* Initialize field contents to default value */
            if (PB_LTYPE(iter.pos->type) == PB_LTYPE_SUBMESSAGE)
            {
+                /* Initialize submessage to defaults */
                pb_message_set_to_defaults((const pb_field_t *) iter.pos->ptr, iter.pData);
            }
            else if (iter.pos->ptr != NULL)
            {
+                /* Initialize to default value */
                memcpy(iter.pData, iter.pos->ptr, iter.pos->data_size);
            }
            else
            {
+                /* Initialize to zeros */
                memset(iter.pData, 0, iter.pos->data_size);
            }
        }
+        else if (PB_ATYPE(type) == PB_ATYPE_POINTER)
+        {
+            /* Initialize the pointer to NULL. */
+            *(void**)iter.pData = NULL;
+            
+            /* Initialize array count to 0. */
+            if (PB_HTYPE(type) == PB_HTYPE_REPEATED)
+            {
+                *(size_t*)iter.pSize = 0;
+            }
+        }
        else if (PB_ATYPE(type) == PB_ATYPE_CALLBACK)
        {
-            continue; /* Don't overwrite callback */
+            /* Don't overwrite callback */
        }
    } while (pb_field_next(&iter));
 }
@@ -646,7 +836,7 @@ static void pb_message_set_to_defaults(const pb_field_t fields[], void *dest_str

 bool checkreturn pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct)
 {
-    uint8_t fields_seen[(PB_MAX_REQUIRED_FIELDS + 7) / 8] = {0}; /* Used to check for required fields */
+    uint8_t fields_seen[(PB_MAX_REQUIRED_FIELDS + 7) / 8] = {0, 0, 0, 0, 0, 0, 0, 0};
    uint32_t extension_range_start = 0;
    pb_field_iterator_t iter;
    
@@ -722,7 +912,7 @@ bool checkreturn pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[
        } while (pb_field_next(&iter));
        
        /* Fixup if last field was also required. */
-        if (PB_HTYPE(last_type) == PB_HTYPE_REQUIRED && iter.pos->tag)
+        if (PB_HTYPE(last_type) == PB_HTYPE_REQUIRED && iter.pos->tag != 0)
            req_field_count++;
        
        /* Check the whole bytes */
@@ -742,8 +932,16 @@ bool checkreturn pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[

 bool checkreturn pb_decode(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct)
 {
+    bool status;
    pb_message_set_to_defaults(fields, dest_struct);
-    return pb_decode_noinit(stream, fields, dest_struct);
+    status = pb_decode_noinit(stream, fields, dest_struct);
+    
+#ifdef PB_ENABLE_MALLOC
+    if (!status)
+        pb_release(fields, dest_struct);
+#endif
+    
+    return status;
 }

 bool pb_decode_delimited(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct)
@@ -759,6 +957,96 @@ bool pb_decode_delimited(pb_istream_t *stream, const pb_field_t fields[], void *
    return status;
 }

+#ifdef PB_ENABLE_MALLOC
+static void pb_release_single_field(const pb_field_iterator_t *iter)
+{
+    pb_type_t type;
+    type = iter->pos->type;
+
+    /* Release anything contained inside an extension or submsg.
+     * This has to be done even if the submsg itself is statically
+     * allocated. */
+    if (PB_LTYPE(type) == PB_LTYPE_EXTENSION)
+    {
+        /* Release fields from all extensions in the linked list */
+        pb_extension_t *ext = *(pb_extension_t**)iter->pData;
+        while (ext != NULL)
+        {
+            pb_field_iterator_t ext_iter;
+            iter_from_extension(&ext_iter, ext);
+            pb_release_single_field(&ext_iter);
+            ext = ext->next;
+        }
+    }
+    else if (PB_LTYPE(type) == PB_LTYPE_SUBMESSAGE)
+    {
+        /* Release fields in submessage or submsg array */
+        void *pItem = iter->pData;
+        pb_size_t count = 1;
+        
+        if (PB_ATYPE(type) == PB_ATYPE_POINTER)
+        {
+            pItem = *(void**)iter->pData;
+        }
+        
+        if (PB_HTYPE(type) == PB_HTYPE_REPEATED)
+        {
+            count = *(pb_size_t*)iter->pSize;
+        }
+        
+        if (pItem)
+        {
+            while (count--)
+            {
+                pb_release((const pb_field_t*)iter->pos->ptr, pItem);
+                pItem = (uint8_t*)pItem + iter->pos->data_size;
+            }
+        }
+    }
+    
+    if (PB_ATYPE(type) == PB_ATYPE_POINTER)
+    {
+        if (PB_HTYPE(type) == PB_HTYPE_REPEATED &&
+            (PB_LTYPE(type) == PB_LTYPE_STRING ||
+             PB_LTYPE(type) == PB_LTYPE_BYTES))
+        {
+            /* Release entries in repeated string or bytes array */
+            void **pItem = *(void***)iter->pData;
+            pb_size_t count = *(pb_size_t*)iter->pSize;
+            while (count--)
+            {
+                pb_free(*pItem);
+                *pItem++ = NULL;
+            }
+        }
+        
+        if (PB_HTYPE(type) == PB_HTYPE_REPEATED)
+        {
+            /* We are going to release the array, so set the size to 0 */
+            *(pb_size_t*)iter->pSize = 0;
+        }
+        
+        /* Release main item */
+        pb_free(*(void**)iter->pData);
+        *(void**)iter->pData = NULL;
+    }
+}
+
+void pb_release(const pb_field_t fields[], void *dest_struct)
+{
+    pb_field_iterator_t iter;
+    pb_field_init(&iter, fields, dest_struct);
+
+    if (iter.pos->tag == 0)
+        return; /* Empty message type */
+    
+    do
+    {
+        pb_release_single_field(&iter);
+    } while (pb_field_next(&iter));
+}
+#endif
+
 /* Field decoders */

 bool pb_decode_svarint(pb_istream_t *stream, int64_t *dest)
@@ -817,7 +1105,7 @@ bool pb_decode_fixed64(pb_istream_t *stream, void *dest)
    #endif   
 }

-bool checkreturn pb_dec_varint(pb_istream_t *stream, const pb_field_t *field, void *dest)
+static bool checkreturn pb_dec_varint(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
    uint64_t value;
    if (!pb_decode_varint(stream, &value))
@@ -835,7 +1123,7 @@ bool checkreturn pb_dec_varint(pb_istream_t *stream, const pb_field_t *field, vo
    return true;
 }

-bool checkreturn pb_dec_uvarint(pb_istream_t *stream, const pb_field_t *field, void *dest)
+static bool checkreturn pb_dec_uvarint(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
    uint64_t value;
    if (!pb_decode_varint(stream, &value))
@@ -851,7 +1139,7 @@ bool checkreturn pb_dec_uvarint(pb_istream_t *stream, const pb_field_t *field, v
    return true;
 }

-bool checkreturn pb_dec_svarint(pb_istream_t *stream, const pb_field_t *field, void *dest)
+static bool checkreturn pb_dec_svarint(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
    int64_t value;
    if (!pb_decode_svarint(stream, &value))
@@ -867,51 +1155,89 @@ bool checkreturn pb_dec_svarint(pb_istream_t *stream, const pb_field_t *field, v
    return true;
 }

-bool checkreturn pb_dec_fixed32(pb_istream_t *stream, const pb_field_t *field, void *dest)
+static bool checkreturn pb_dec_fixed32(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
    UNUSED(field);
    return pb_decode_fixed32(stream, dest);
 }

-bool checkreturn pb_dec_fixed64(pb_istream_t *stream, const pb_field_t *field, void *dest)
+static bool checkreturn pb_dec_fixed64(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
    UNUSED(field);
    return pb_decode_fixed64(stream, dest);
 }

-bool checkreturn pb_dec_bytes(pb_istream_t *stream, const pb_field_t *field, void *dest)
-{
-    pb_bytes_array_t *x = (pb_bytes_array_t*)dest;
-    
-    uint32_t temp;
-    if (!pb_decode_varint32(stream, &temp))
-        return false;
-    x->size = temp;
-    
-    /* Check length, noting the space taken by the size_t header. */
-    if (x->size > field->data_size - offsetof(pb_bytes_array_t, bytes))
-        PB_RETURN_ERROR(stream, "bytes overflow");
-    
-    return pb_read(stream, x->bytes, x->size);
-}
-
-bool checkreturn pb_dec_string(pb_istream_t *stream, const pb_field_t *field, void *dest)
+static bool checkreturn pb_dec_bytes(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
    uint32_t size;
+    size_t alloc_size;
+    pb_bytes_array_t *bdest;
+    
+    if (!pb_decode_varint32(stream, &size))
+        return false;
+    
+    alloc_size = PB_BYTES_ARRAY_T_ALLOCSIZE(size);
+    if (size > alloc_size)
+        PB_RETURN_ERROR(stream, "size too large");
+    
+    if (PB_ATYPE(field->type) == PB_ATYPE_POINTER)
+    {
+#ifndef PB_ENABLE_MALLOC
+        PB_RETURN_ERROR(stream, "no malloc support");
+#else
+        if (!allocate_field(stream, dest, alloc_size, 1))
+            return false;
+        bdest = *(pb_bytes_array_t**)dest;
+#endif
+    }
+    else
+    {
+        if (alloc_size > field->data_size)
+            PB_RETURN_ERROR(stream, "bytes overflow");
+        bdest = (pb_bytes_array_t*)dest;
+    }
+    
+    bdest->size = size;
+
+    return pb_read(stream, bdest->bytes, size);
+}
+
+static bool checkreturn pb_dec_string(pb_istream_t *stream, const pb_field_t *field, void *dest)
+{
+    uint32_t size;
+    size_t alloc_size;
    bool status;
    if (!pb_decode_varint32(stream, &size))
        return false;
    
-    /* Check length, noting the null terminator */
-    if (size + 1 > field->data_size)
+    /* Space for null terminator */
+    alloc_size = size + 1;
+    
+    if (alloc_size < size)
+        PB_RETURN_ERROR(stream, "size too large");
+    
+    if (PB_ATYPE(field->type) == PB_ATYPE_POINTER)
+    {
+#ifndef PB_ENABLE_MALLOC
+        PB_RETURN_ERROR(stream, "no malloc support");
+#else
+        if (!allocate_field(stream, dest, alloc_size, 1))
+            return false;
+        dest = *(void**)dest;
+#endif
+    }
+    else
+    {
+        if (alloc_size > field->data_size)
            PB_RETURN_ERROR(stream, "string overflow");
+    }
    
    status = pb_read(stream, (uint8_t*)dest, size);
    *((uint8_t*)dest + size) = 0;
    return status;
 }

-bool checkreturn pb_dec_submessage(pb_istream_t *stream, const pb_field_t *field, void *dest)
+static bool checkreturn pb_dec_submessage(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
    bool status;
    pb_istream_t substream;
--- a/pb_decode.h
+++ b/pb_decode.h
@@ -73,6 +73,9 @@ bool pb_decode(pb_istream_t *stream, const pb_field_t fields[], void *dest_struc
 *
 * This can also be used for 'merging' two messages, i.e. update only the
 * fields that exist in the new message.
+ *
+ * Note: If this function returns with an error, it will not release any
+ * dynamically allocated fields. You will need to call pb_release() yourself.
 */
 bool pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct);

@@ -82,6 +85,14 @@ bool pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[], void *des
 */
 bool pb_decode_delimited(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct);

+#ifdef PB_ENABLE_MALLOC
+/* Release any allocated pointer fields. If you use dynamic allocation, you should
+ * call this for any successfully decoded message when you are done with it. If
+ * pb_decode() returns with an error, the message is already released.
+ */
+void pb_release(const pb_field_t fields[], void *dest_struct);
+#endif
+

 /**************************************
 * Functions for manipulating streams *
--- a/pb_encode.c
+++ b/pb_encode.c
@@ -3,7 +3,6 @@
 * 2011 Petteri Aimonen <jpa@kapsi.fi>
 */

-#define NANOPB_INTERNALS
 #include "pb.h"
 #include "pb_encode.h"

@@ -174,11 +173,12 @@ static bool checkreturn encode_array(pb_ostream_t *stream, const pb_field_t *fie
                return false;

            /* Normally the data is stored directly in the array entries, but
-             * for pointer-type string fields, the array entries are actually
-             * string pointers. So we have to dereference once more to get to
-             * the character data. */
+             * for pointer-type string and bytes fields, the array entries are
+             * actually pointers themselves also. So we have to dereference once
+             * more to get to the actual data. */
            if (PB_ATYPE(field->type) == PB_ATYPE_POINTER &&
-                PB_LTYPE(field->type) == PB_LTYPE_STRING)
+                (PB_LTYPE(field->type) == PB_LTYPE_STRING ||
+                 PB_LTYPE(field->type) == PB_LTYPE_BYTES))
            {
                if (!func(stream, field, *(const void* const*)p))
                    return false;      
@@ -378,6 +378,17 @@ bool pb_encode_delimited(pb_ostream_t *stream, const pb_field_t fields[], const
    return pb_encode_submessage(stream, fields, src_struct);
 }

+bool pb_get_encoded_size(size_t *size, const pb_field_t fields[], const void *src_struct)
+{
+    pb_ostream_t stream = PB_OSTREAM_SIZING;
+    
+    if (!pb_encode(&stream, fields, src_struct))
+        return false;
+    
+    *size = stream.bytes_written;
+    return true;
+}
+
 /********************
 * Helper functions *
 ********************/
@@ -404,9 +415,9 @@ bool checkreturn pb_encode_svarint(pb_ostream_t *stream, int64_t value)
 {
    uint64_t zigzagged;
    if (value < 0)
-        zigzagged = (uint64_t)(~(value << 1));
+        zigzagged = ~((uint64_t)value << 1);
    else
-        zigzagged = (uint64_t)(value << 1);
+        zigzagged = (uint64_t)value << 1;
    
    return pb_encode_varint(stream, zigzagged);
 }
@@ -447,7 +458,7 @@ bool checkreturn pb_encode_fixed64(pb_ostream_t *stream, const void *value)

 bool checkreturn pb_encode_tag(pb_ostream_t *stream, pb_wire_type_t wiretype, uint32_t field_number)
 {
-    uint64_t tag = wiretype | (field_number << 3);
+    uint64_t tag = ((uint64_t)field_number << 3) | wiretype;
    return pb_encode_varint(stream, tag);
 }

@@ -543,7 +554,7 @@ bool checkreturn pb_encode_submessage(pb_ostream_t *stream, const pb_field_t fie

 /* Field encoders */

-bool checkreturn pb_enc_varint(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_varint(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
    int64_t value = 0;
    
@@ -561,7 +572,7 @@ bool checkreturn pb_enc_varint(pb_ostream_t *stream, const pb_field_t *field, co
    return pb_encode_varint(stream, (uint64_t)value);
 }

-bool checkreturn pb_enc_uvarint(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_uvarint(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
    uint64_t value = 0;
    
@@ -575,7 +586,7 @@ bool checkreturn pb_enc_uvarint(pb_ostream_t *stream, const pb_field_t *field, c
    return pb_encode_varint(stream, value);
 }

-bool checkreturn pb_enc_svarint(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_svarint(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
    int64_t value = 0;
    
@@ -589,36 +600,38 @@ bool checkreturn pb_enc_svarint(pb_ostream_t *stream, const pb_field_t *field, c
    return pb_encode_svarint(stream, value);
 }

-bool checkreturn pb_enc_fixed64(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_fixed64(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
    UNUSED(field);
    return pb_encode_fixed64(stream, src);
 }

-bool checkreturn pb_enc_fixed32(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_fixed32(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
    UNUSED(field);
    return pb_encode_fixed32(stream, src);
 }

-bool checkreturn pb_enc_bytes(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_bytes(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
-    if (PB_ATYPE(field->type) == PB_ATYPE_POINTER)
-    {
-        const pb_bytes_ptr_t *bytes = (const pb_bytes_ptr_t*)src;
-        return pb_encode_string(stream, bytes->bytes, bytes->size);
-    }
-    else
-    {
    const pb_bytes_array_t *bytes = (const pb_bytes_array_t*)src;
-        if (bytes->size + offsetof(pb_bytes_array_t, bytes) > field->data_size)
+    
+    if (src == NULL)
+    {
+        /* Threat null pointer as an empty bytes field */
+        return pb_encode_string(stream, NULL, 0);
+    }
+    
+    if (PB_ATYPE(field->type) == PB_ATYPE_STATIC &&
+        PB_BYTES_ARRAY_T_ALLOCSIZE(bytes->size) > field->data_size)
+    {
        PB_RETURN_ERROR(stream, "bytes size exceeded");
+    }
    
    return pb_encode_string(stream, bytes->bytes, bytes->size);
-    }
 }

-bool checkreturn pb_enc_string(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_string(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
    /* strnlen() is not always available, so just use a loop */
    size_t size = 0;
@@ -628,16 +641,23 @@ bool checkreturn pb_enc_string(pb_ostream_t *stream, const pb_field_t *field, co
    if (PB_ATYPE(field->type) == PB_ATYPE_POINTER)
        max_size = (size_t)-1;

+    if (src == NULL)
+    {
+        size = 0; /* Threat null pointer as an empty string */
+    }
+    else
+    {
        while (size < max_size && *p != '\0')
        {
            size++;
            p++;
        }
+    }

    return pb_encode_string(stream, (const uint8_t*)src, size);
 }

-bool checkreturn pb_enc_submessage(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_submessage(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
    if (field->ptr == NULL)
        PB_RETURN_ERROR(stream, "invalid field descriptor");
--- a/pb_encode.h
+++ b/pb_encode.h
@@ -71,6 +71,10 @@ bool pb_encode(pb_ostream_t *stream, const pb_field_t fields[], const void *src_
 */
 bool pb_encode_delimited(pb_ostream_t *stream, const pb_field_t fields[], const void *src_struct);

+/* Encode the message to get the size of the encoded data, but do not store
+ * the data. */
+bool pb_get_encoded_size(size_t *size, const pb_field_t fields[], const void *src_struct);
+
 /**************************************
 * Functions for manipulating streams *
 **************************************/
--- a/tests/Makefile
+++ b/tests/Makefile
@@ -4,3 +4,18 @@ all:
 clean:
 	scons -c

+coverage:
+	rm -rf build coverage
+
+	# LCOV does not like the newer gcov format
+	scons CC=gcc-4.6 CXX=gcc-4.6
+
+	# We are only interested in pb_encode.o and pb_decode.o
+	find build -name '*.gcda' -and \! \( -name '*pb_encode*' -or -name '*pb_decode*' \) -exec rm '{}' \;
+
+	# Collect the data
+	mkdir build/coverage
+	lcov --base-directory . --directory build/ --gcov-tool gcov-4.6 -c -o build/coverage/nanopb.info
+
+	# Generate HTML
+	genhtml -o build/coverage build/coverage/nanopb.info
--- a/tests/SConstruct
+++ b/tests/SConstruct
@@ -19,8 +19,8 @@ env = Environment(ENV = os.environ, tools = ['default', 'nanopb'])
 # Allow overriding the compiler with scons CC=???
 if 'CC' in ARGUMENTS: env.Replace(CC = ARGUMENTS['CC'])
 if 'CXX' in ARGUMENTS: env.Replace(CXX = ARGUMENTS['CXX'])
-if 'CFLAGS' in ARGUMENTS: env.Append(CCFLAGS = ARGUMENTS['CFLAGS'])
-if 'CXXFLAGS' in ARGUMENTS: env.Append(CCFLAGS = ARGUMENTS['CXXFLAGS'])
+if 'CCFLAGS' in ARGUMENTS: env.Append(CCFLAGS = ARGUMENTS['CCFLAGS'])
+if 'CXXFLAGS' in ARGUMENTS: env.Append(CXXFLAGS = ARGUMENTS['CXXFLAGS'])

 # Add the builders defined in site_init.py
 add_nanopb_builders(env)
@@ -33,13 +33,16 @@ env.Append(PROTOCPATH = '#../generator')

 # Check the compilation environment, unless we are just cleaning up.
 if not env.GetOption('clean'):
-    def check_ccflags(context, flags):
+    def check_ccflags(context, flags, linkflags = ''):
        '''Check if given CCFLAGS are supported'''
        context.Message('Checking support for CCFLAGS="%s"... ' % flags)
        oldflags = context.env['CCFLAGS']
+        oldlinkflags = context.env['CCFLAGS']
        context.env.Append(CCFLAGS = flags)
+        context.env.Append(LINKFLAGS = linkflags)
        result = context.TryCompile("int main() {return 0;}", '.c')
        context.env.Replace(CCFLAGS = oldflags)
+        context.env.Replace(LINKFLAGS = oldlinkflags)
        context.Result(result)
        return result
    
@@ -50,14 +53,17 @@ if not env.GetOption('clean'):
    stdint = conf.CheckCHeader('stdint.h')
    stddef = conf.CheckCHeader('stddef.h')
    string = conf.CheckCHeader('string.h')
+    stdlib = conf.CheckCHeader('stdlib.h')
    if not stdbool or not stdint or not stddef or not string:
        conf.env.Append(CPPDEFINES = {'PB_SYSTEM_HEADER': '\\"pb_syshdr.h\\"'})
        conf.env.Append(CPPPATH = "#../extra")
+	conf.env.Append(SYSHDR = '\\"pb_syshdr.h\\"')
        
        if stdbool: conf.env.Append(CPPDEFINES = {'HAVE_STDBOOL_H': 1})
        if stdint: conf.env.Append(CPPDEFINES = {'HAVE_STDINT_H': 1})
        if stddef: conf.env.Append(CPPDEFINES = {'HAVE_STDDEF_H': 1})
        if string: conf.env.Append(CPPDEFINES = {'HAVE_STRING_H': 1})
+        if stdlib: conf.env.Append(CPPDEFINES = {'HAVE_STDLIB_H': 1})
    
    # Check if we can use pkg-config to find protobuf include path
    status, output = conf.TryAction('pkg-config protobuf --variable=includedir > $TARGET')
@@ -70,16 +76,24 @@ if not env.GetOption('clean'):
    if 'gcc' in env['CC']:
        if conf.CheckLib('mudflap'):
            conf.env.Append(CCFLAGS = '-fmudflap')
-            conf.env.Append(LINKFLAGS = '-lmudflap -fmudflap')
+            conf.env.Append(LINKFLAGS = '-fmudflap')
    
    # Check if we can use extra strict warning flags (only with GCC)
    extra = '-Wcast-qual -Wlogical-op -Wconversion'
+    extra += ' -fstrict-aliasing -Wstrict-aliasing=1'
    extra += ' -Wmissing-prototypes -Wmissing-declarations -Wredundant-decls'
    extra += ' -Wstack-protector '
    if 'gcc' in env['CC']:
        if conf.CheckCCFLAGS(extra):
            conf.env.Append(CORECFLAGS = extra)
    
+    # Check if we can use undefined behaviour sanitizer (only with clang)
+    extra = '-fsanitize=undefined '
+    if 'clang' in env['CC']:
+        if conf.CheckCCFLAGS(extra, linkflags = extra):
+            conf.env.Append(CORECFLAGS = extra)
+            conf.env.Append(LINKFLAGS = extra)
+    
    # End the config stuff
    env = conf.Finish()

@@ -88,15 +102,15 @@ if 'gcc' in env['CC']:
    # GNU Compiler Collection
    
    # Debug info, warnings as errors
-    env.Append(CFLAGS = '-ansi -pedantic -g -O0 -Wall -Werror --coverage -fstack-protector-all')
+    env.Append(CFLAGS = '-ansi -pedantic -g -Wall -Werror -fprofile-arcs -ftest-coverage ')
    env.Append(CORECFLAGS = '-Wextra')
-    env.Append(LINKFLAGS = '--coverage')
+    env.Append(LINKFLAGS = '-g --coverage')
    
    # We currently need uint64_t anyway, even though ANSI C90 otherwise..
    env.Append(CFLAGS = '-Wno-long-long')
 elif 'clang' in env['CC']:
    # CLang
-    env.Append(CFLAGS = '-ansi -g -O0 -Wall -Werror')
+    env.Append(CFLAGS = '-ansi -g -Wall -Werror')
    env.Append(CORECFLAGS = ' -Wextra -Wcast-qual -Wconversion')
 elif 'cl' in env['CC']:
    # Microsoft Visual C++
@@ -116,10 +130,10 @@ elif 'tcc' in env['CC']:

 env.SetDefault(CORECFLAGS = '')

-if 'clang++' in env['CXX']:
-    env.Append(CXXFLAGS = '-g -O0 -Wall -Werror -Wextra -Wno-missing-field-initializers')
-elif 'g++' in env['CXX']:
-    env.Append(CXXFLAGS = '-g -O0 -Wall -Werror -Wextra -Wno-missing-field-initializers')
+if 'clang' in env['CXX']:
+    env.Append(CXXFLAGS = '-g -Wall -Werror -Wextra -Wno-missing-field-initializers')
+elif 'g++' in env['CXX'] or 'gcc' in env['CXX']:
+    env.Append(CXXFLAGS = '-g -Wall -Werror -Wextra -Wno-missing-field-initializers')
 elif 'cl' in env['CXX']:
    env.Append(CXXFLAGS = '/Zi /W2 /WX')

@@ -128,6 +142,12 @@ import os.path
 env['VARIANT_DIR'] = 'build'
 env['BUILD'] = '#' + env['VARIANT_DIR']
 env['COMMON'] = '#' + env['VARIANT_DIR'] + '/common'
-for subdir in Glob('*/SConscript'):
+
+# Include common/SConscript first to make sure its exports are available
+# to other SConscripts.
+SConscript("common/SConscript", exports = 'env', variant_dir = env['VARIANT_DIR'] + '/common')
+
+for subdir in Glob('*/SConscript') + Glob('regression/*/SConscript'):
+    if str(subdir).startswith("common"): continue
    SConscript(subdir, exports = 'env', variant_dir = env['VARIANT_DIR'] + '/' + os.path.dirname(str(subdir)))

--- a/tests/alltypes/alltypes.proto
+++ b/tests/alltypes/alltypes.proto
@@ -14,16 +14,16 @@ enum HugeEnum {
 }

 message Limits {
-    required int32      int32_min  =  1;
-    required int32      int32_max  =  2;
-    required uint32     uint32_min =  3;
-    required uint32     uint32_max =  4;
-    required int64      int64_min  =  5;
-    required int64      int64_max  =  6;
-    required uint64     uint64_min =  7;
-    required uint64     uint64_max =  8;
-    required HugeEnum   enum_min   =  9;
-    required HugeEnum   enum_max   = 10;
+    required int32      int32_min  =  1 [default = 2147483647];
+    required int32      int32_max  =  2 [default = -2147483647];
+    required uint32     uint32_min =  3 [default = 4294967295];
+    required uint32     uint32_max =  4 [default = 0];
+    required int64      int64_min  =  5 [default = 9223372036854775807];
+    required int64      int64_max  =  6 [default = -9223372036854775807];
+    required uint64     uint64_min =  7 [default = 18446744073709551615];
+    required uint64     uint64_max =  8 [default = 0];
+    required HugeEnum   enum_min   =  9 [default = Positive];
+    required HugeEnum   enum_max   = 10 [default = Negative];
 }

 enum MyEnum {
--- a/tests/alltypes/decode_alltypes.c
+++ b/tests/alltypes/decode_alltypes.c
@@ -19,10 +19,12 @@
   the decoding and checks the fields. */
 bool check_alltypes(pb_istream_t *stream, int mode)
 {
-    AllTypes alltypes;
+    /* Uses _init_default to just make sure that it works. */
+    AllTypes alltypes = AllTypes_init_default;
    
    /* Fill with garbage to better detect initialization errors */
    memset(&alltypes, 0xAA, sizeof(alltypes));
+    alltypes.extensions = 0;
    
    if (!pb_decode(stream, AllTypes_fields, &alltypes))
        return false;
--- a/tests/alltypes/encode_alltypes.c
+++ b/tests/alltypes/encode_alltypes.c
@@ -13,7 +13,7 @@ int main(int argc, char **argv)
    int mode = (argc > 1) ? atoi(argv[1]) : 0;
    
    /* Initialize the structure with constants */
-    AllTypes alltypes = {0};
+    AllTypes alltypes = AllTypes_init_zero;
    
    alltypes.req_int32         = -1001;
    alltypes.req_int64         = -1002;
--- a/tests/alltypes_callback/SConscript
+++ b/tests/alltypes_callback/SConscript
@@ -0,0 +1,23 @@
+# Test the AllTypes encoding & decoding using callbacks for all fields.
+
+Import("env")
+
+c = Copy("$TARGET", "$SOURCE")
+env.Command("alltypes.proto", "#alltypes/alltypes.proto", c)
+
+env.NanopbProto(["alltypes", "alltypes.options"])
+enc = env.Program(["encode_alltypes_callback.c", "alltypes.pb.c", "$COMMON/pb_encode.o"])
+dec = env.Program(["decode_alltypes_callback.c", "alltypes.pb.c", "$COMMON/pb_decode.o"])
+
+refdec = "$BUILD/alltypes/decode_alltypes$PROGSUFFIX"
+
+# Encode and compare results
+env.RunTest(enc)
+env.RunTest("decode_alltypes.output", [refdec, "encode_alltypes_callback.output"])
+env.RunTest("decode_alltypes_callback.output", [dec, "encode_alltypes_callback.output"])
+
+# Do the same thing with the optional fields present
+env.RunTest("optionals.output", enc, ARGS = ['1'])
+env.RunTest("optionals.refdecout", [refdec, "optionals.output"], ARGS = ['1'])
+env.RunTest("optionals.decout", [dec, "optionals.output"], ARGS = ['1'])
+
--- a/tests/alltypes_callback/alltypes.options
+++ b/tests/alltypes_callback/alltypes.options
@@ -0,0 +1,3 @@
+# Generate all fields as callbacks.
+AllTypes.*              type:FT_CALLBACK
+SubMessage.substuff1    max_size:16
--- a/tests/alltypes_callback/decode_alltypes_callback.c
+++ b/tests/alltypes_callback/decode_alltypes_callback.c
@@ -0,0 +1,424 @@
+/* Attempts to test all the datatypes supported by ProtoBuf when used as callback fields.
+ * Note that normally there would be no reason to use callback fields for this,
+ * because each encoder defined here only gives a single field.
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdlib.h>
+#include <pb_decode.h>
+#include "alltypes.pb.h"
+#include "test_helpers.h"
+
+#define TEST(x) if (!(x)) { \
+    printf("Test " #x " failed (in field %d).\n", field->tag); \
+    return false; \
+    }
+
+static bool read_varint(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    uint64_t value;
+    if (!pb_decode_varint(stream, &value))
+        return false;
+    
+    TEST((int64_t)value == (long)*arg);
+    return true;
+}
+
+static bool read_svarint(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    int64_t value;
+    if (!pb_decode_svarint(stream, &value))
+        return false;
+    
+    TEST(value == (long)*arg);
+    return true;
+}
+
+static bool read_fixed32(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    uint32_t value;
+    if (!pb_decode_fixed32(stream, &value))
+        return false;
+    
+    TEST(value == *(uint32_t*)*arg);
+    return true;
+}
+
+static bool read_fixed64(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    uint64_t value;
+    if (!pb_decode_fixed64(stream, &value))
+        return false;
+    
+    TEST(value == *(uint64_t*)*arg);
+    return true;
+}
+
+static bool read_string(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    uint8_t buf[16] = {0};
+    size_t len = stream->bytes_left;
+    
+    if (len > sizeof(buf) - 1 || !pb_read(stream, buf, len))
+        return false;
+    
+    TEST(strcmp((char*)buf, *arg) == 0);
+    return true;
+}
+
+static bool read_submsg(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    SubMessage submsg = {""};
+    
+    if (!pb_decode(stream, SubMessage_fields, &submsg))
+        return false;
+    
+    TEST(memcmp(&submsg, *arg, sizeof(submsg)));
+    return true;
+}
+
+static bool read_emptymsg(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    EmptyMessage emptymsg = {0};
+    return pb_decode(stream, EmptyMessage_fields, &emptymsg);
+}
+
+static bool read_repeated_varint(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    int32_t** expected = (int32_t**)arg;
+    uint64_t value;
+    if (!pb_decode_varint(stream, &value))
+        return false;
+
+    TEST(*(*expected)++ == value);
+    return true;
+}
+
+static bool read_repeated_svarint(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    int32_t** expected = (int32_t**)arg;
+    int64_t value;
+    if (!pb_decode_svarint(stream, &value))
+        return false;
+
+    TEST(*(*expected)++ == value);
+    return true;
+}
+
+static bool read_repeated_fixed32(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    uint32_t** expected = (uint32_t**)arg;
+    uint32_t value;
+    if (!pb_decode_fixed32(stream, &value))
+        return false;
+
+    TEST(*(*expected)++ == value);
+    return true;
+}
+
+static bool read_repeated_fixed64(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    uint64_t** expected = (uint64_t**)arg;
+    uint64_t value;
+    if (!pb_decode_fixed64(stream, &value))
+        return false;
+
+    TEST(*(*expected)++ == value);
+    return true;
+}
+
+static bool read_repeated_string(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    uint8_t*** expected = (uint8_t***)arg;
+    uint8_t buf[16] = {0};
+    size_t len = stream->bytes_left;
+    
+    if (len > sizeof(buf) - 1 || !pb_read(stream, buf, len))
+        return false;
+    
+    TEST(strcmp((char*)*(*expected)++, (char*)buf) == 0);
+    return true;
+}
+
+static bool read_repeated_submsg(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    SubMessage** expected = (SubMessage**)arg;
+    SubMessage decoded = {""};
+    if (!pb_decode(stream, SubMessage_fields, &decoded))
+        return false;
+
+    TEST(memcmp((*expected)++, &decoded, sizeof(decoded)) == 0);
+    return true;
+}
+
+static bool read_limits(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    Limits decoded = {0};
+    if (!pb_decode(stream, Limits_fields, &decoded))
+        return false;
+
+    TEST(decoded.int32_min  == INT32_MIN);
+    TEST(decoded.int32_max  == INT32_MAX);
+    TEST(decoded.uint32_min == 0);
+    TEST(decoded.uint32_max == UINT32_MAX);
+    TEST(decoded.int64_min  == INT64_MIN);
+    TEST(decoded.int64_max  == INT64_MAX);
+    TEST(decoded.uint64_min == 0);
+    TEST(decoded.uint64_max == UINT64_MAX);
+    TEST(decoded.enum_min   == HugeEnum_Negative);
+    TEST(decoded.enum_max   == HugeEnum_Positive);
+    
+    return true;
+}
+
+/* This function is called once from main(), it handles
+   the decoding and checks the fields. */
+bool check_alltypes(pb_istream_t *stream, int mode)
+{
+    /* Values for use from callbacks through pointers. */
+    uint32_t    req_fixed32     = 1008;
+    int32_t     req_sfixed32    = -1009;
+    float       req_float       = 1010.0f;
+    uint64_t    req_fixed64     = 1011;
+    int64_t     req_sfixed64    = -1012;
+    double      req_double      = 1013.0;
+    SubMessage  req_submsg      = {"1016", 1016};
+    
+    int32_t     rep_int32[5]    = {0, 0, 0, 0, -2001};
+    int32_t     rep_int64[5]    = {0, 0, 0, 0, -2002};
+    int32_t     rep_uint32[5]   = {0, 0, 0, 0,  2003};
+    int32_t     rep_uint64[5]   = {0, 0, 0, 0,  2004};
+    int32_t     rep_sint32[5]   = {0, 0, 0, 0, -2005};
+    int32_t     rep_sint64[5]   = {0, 0, 0, 0, -2006};
+    int32_t     rep_bool[5]     = {false, false, false, false, true};
+    uint32_t    rep_fixed32[5]  = {0, 0, 0, 0,  2008};
+    int32_t     rep_sfixed32[5] = {0, 0, 0, 0, -2009};
+    float       rep_float[5]    = {0, 0, 0, 0,  2010.0f};
+    uint64_t    rep_fixed64[5]  = {0, 0, 0, 0,  2011};
+    int64_t     rep_sfixed64[5] = {0, 0, 0, 0, -2012};
+    double      rep_double[5]   = {0, 0, 0, 0,  2013.0};
+    char*       rep_string[5]   = {"", "", "", "", "2014"};
+    char*       rep_bytes[5]    = {"", "", "", "", "2015"};
+    SubMessage  rep_submsg[5]   = {{"", 0, 0, 3},
+                                   {"", 0, 0, 3},
+                                   {"", 0, 0, 3},
+                                   {"", 0, 0, 3},
+                                   {"2016", 2016, true, 2016}};
+    int32_t     rep_enum[5]     = {0, 0, 0, 0, MyEnum_Truth};
+    
+    uint32_t    opt_fixed32     = 3048;
+    int32_t     opt_sfixed32    = 3049;
+    float       opt_float       = 3050.0f;
+    uint64_t    opt_fixed64     = 3051;
+    int64_t     opt_sfixed64    = 3052;
+    double      opt_double      = 3053.0f;
+    SubMessage  opt_submsg      = {"3056", 3056};
+    
+    /* Bind callbacks for required fields */
+    AllTypes alltypes;
+    
+    /* Fill with garbage to better detect initialization errors */
+    memset(&alltypes, 0xAA, sizeof(alltypes));
+    alltypes.extensions = 0;
+    
+    alltypes.req_int32.funcs.decode = &read_varint;
+    alltypes.req_int32.arg = (void*)-1001;
+    
+    alltypes.req_int64.funcs.decode = &read_varint;
+    alltypes.req_int64.arg = (void*)-1002;
+    
+    alltypes.req_uint32.funcs.decode = &read_varint;
+    alltypes.req_uint32.arg = (void*)1003;
+
+    alltypes.req_uint32.funcs.decode = &read_varint;
+    alltypes.req_uint32.arg = (void*)1003;    
+    
+    alltypes.req_uint64.funcs.decode = &read_varint;
+    alltypes.req_uint64.arg = (void*)1004;
+    
+    alltypes.req_sint32.funcs.decode = &read_svarint;
+    alltypes.req_sint32.arg = (void*)-1005;   
+    
+    alltypes.req_sint64.funcs.decode = &read_svarint;
+    alltypes.req_sint64.arg = (void*)-1006;   
+    
+    alltypes.req_bool.funcs.decode = &read_varint;
+    alltypes.req_bool.arg = (void*)true;   
+    
+    alltypes.req_fixed32.funcs.decode = &read_fixed32;
+    alltypes.req_fixed32.arg = &req_fixed32;
+    
+    alltypes.req_sfixed32.funcs.decode = &read_fixed32;
+    alltypes.req_sfixed32.arg = &req_sfixed32;
+    
+    alltypes.req_float.funcs.decode = &read_fixed32;
+    alltypes.req_float.arg = &req_float;
+    
+    alltypes.req_fixed64.funcs.decode = &read_fixed64;
+    alltypes.req_fixed64.arg = &req_fixed64;
+    
+    alltypes.req_sfixed64.funcs.decode = &read_fixed64;
+    alltypes.req_sfixed64.arg = &req_sfixed64;
+    
+    alltypes.req_double.funcs.decode = &read_fixed64;
+    alltypes.req_double.arg = &req_double;
+    
+    alltypes.req_string.funcs.decode = &read_string;
+    alltypes.req_string.arg = "1014";
+    
+    alltypes.req_bytes.funcs.decode = &read_string;
+    alltypes.req_bytes.arg = "1015";
+    
+    alltypes.req_submsg.funcs.decode = &read_submsg;
+    alltypes.req_submsg.arg = &req_submsg;
+    
+    alltypes.req_enum.funcs.decode = &read_varint;
+    alltypes.req_enum.arg = (void*)MyEnum_Truth;
+    
+    alltypes.req_emptymsg.funcs.decode = &read_emptymsg;
+    
+    /* Bind callbacks for repeated fields */
+    alltypes.rep_int32.funcs.decode = &read_repeated_varint;
+    alltypes.rep_int32.arg = rep_int32;
+    
+    alltypes.rep_int64.funcs.decode = &read_repeated_varint;
+    alltypes.rep_int64.arg = rep_int64;
+    
+    alltypes.rep_uint32.funcs.decode = &read_repeated_varint;
+    alltypes.rep_uint32.arg = rep_uint32;
+    
+    alltypes.rep_uint64.funcs.decode = &read_repeated_varint;
+    alltypes.rep_uint64.arg = rep_uint64;
+    
+    alltypes.rep_sint32.funcs.decode = &read_repeated_svarint;
+    alltypes.rep_sint32.arg = rep_sint32;
+    
+    alltypes.rep_sint64.funcs.decode = &read_repeated_svarint;
+    alltypes.rep_sint64.arg = rep_sint64;
+    
+    alltypes.rep_bool.funcs.decode = &read_repeated_varint;
+    alltypes.rep_bool.arg = rep_bool;
+    
+    alltypes.rep_fixed32.funcs.decode = &read_repeated_fixed32;
+    alltypes.rep_fixed32.arg = rep_fixed32;
+    
+    alltypes.rep_sfixed32.funcs.decode = &read_repeated_fixed32;
+    alltypes.rep_sfixed32.arg = rep_sfixed32;
+    
+    alltypes.rep_float.funcs.decode = &read_repeated_fixed32;
+    alltypes.rep_float.arg = rep_float;
+    
+    alltypes.rep_fixed64.funcs.decode = &read_repeated_fixed64;
+    alltypes.rep_fixed64.arg = rep_fixed64;
+    
+    alltypes.rep_sfixed64.funcs.decode = &read_repeated_fixed64;
+    alltypes.rep_sfixed64.arg = rep_sfixed64;
+    
+    alltypes.rep_double.funcs.decode = &read_repeated_fixed64;
+    alltypes.rep_double.arg = rep_double;
+    
+    alltypes.rep_string.funcs.decode = &read_repeated_string;
+    alltypes.rep_string.arg = rep_string;
+    
+    alltypes.rep_bytes.funcs.decode = &read_repeated_string;
+    alltypes.rep_bytes.arg = rep_bytes;
+    
+    alltypes.rep_submsg.funcs.decode = &read_repeated_submsg;
+    alltypes.rep_submsg.arg = rep_submsg;
+    
+    alltypes.rep_enum.funcs.decode = &read_repeated_varint;
+    alltypes.rep_enum.arg = rep_enum;
+    
+    alltypes.rep_emptymsg.funcs.decode = &read_emptymsg;
+    
+    alltypes.req_limits.funcs.decode = &read_limits;
+    
+    alltypes.end.funcs.decode = &read_varint;
+    alltypes.end.arg = (void*)1099;
+    
+    /* Bind callbacks for optional fields */
+    if (mode == 1)
+    {
+        alltypes.opt_int32.funcs.decode = &read_varint;
+        alltypes.opt_int32.arg = (void*)3041;
+        
+        alltypes.opt_int64.funcs.decode = &read_varint;
+        alltypes.opt_int64.arg = (void*)3042;
+        
+        alltypes.opt_uint32.funcs.decode = &read_varint;
+        alltypes.opt_uint32.arg = (void*)3043;
+        
+        alltypes.opt_uint64.funcs.decode = &read_varint;
+        alltypes.opt_uint64.arg = (void*)3044;
+        
+        alltypes.opt_sint32.funcs.decode = &read_svarint;
+        alltypes.opt_sint32.arg = (void*)3045;
+        
+        alltypes.opt_sint64.funcs.decode = &read_svarint;
+        alltypes.opt_sint64.arg = (void*)3046;
+        
+        alltypes.opt_bool.funcs.decode = &read_varint;
+        alltypes.opt_bool.arg = (void*)true;
+
+        alltypes.opt_fixed32.funcs.decode = &read_fixed32;
+        alltypes.opt_fixed32.arg = &opt_fixed32;
+        
+        alltypes.opt_sfixed32.funcs.decode = &read_fixed32;
+        alltypes.opt_sfixed32.arg = &opt_sfixed32;
+        
+        alltypes.opt_float.funcs.decode = &read_fixed32;
+        alltypes.opt_float.arg = &opt_float;
+        
+        alltypes.opt_fixed64.funcs.decode = &read_fixed64;
+        alltypes.opt_fixed64.arg = &opt_fixed64;
+        
+        alltypes.opt_sfixed64.funcs.decode = &read_fixed64;
+        alltypes.opt_sfixed64.arg = &opt_sfixed64;
+        
+        alltypes.opt_double.funcs.decode = &read_fixed64;
+        alltypes.opt_double.arg = &opt_double;
+        
+        alltypes.opt_string.funcs.decode = &read_string;
+        alltypes.opt_string.arg = "3054";
+        
+        alltypes.opt_bytes.funcs.decode = &read_string;
+        alltypes.opt_bytes.arg = "3055";
+        
+        alltypes.opt_submsg.funcs.decode = &read_submsg;
+        alltypes.opt_submsg.arg = &opt_submsg;
+        
+        alltypes.opt_enum.funcs.decode = &read_varint;
+        alltypes.opt_enum.arg = (void*)MyEnum_Truth;
+        
+        alltypes.opt_emptymsg.funcs.decode = &read_emptymsg;
+    }
+    
+    return pb_decode(stream, AllTypes_fields, &alltypes);
+}
+
+int main(int argc, char **argv)
+{
+    uint8_t buffer[1024];
+    size_t count;
+    pb_istream_t stream;
+
+    /* Whether to expect the optional values or the default values. */
+    int mode = (argc > 1) ? atoi(argv[1]) : 0;
+    
+    /* Read the data into buffer */
+    SET_BINARY_MODE(stdin);
+    count = fread(buffer, 1, sizeof(buffer), stdin);
+    
+    /* Construct a pb_istream_t for reading from the buffer */
+    stream = pb_istream_from_buffer(buffer, count);
+    
+    /* Decode and print out the stuff */
+    if (!check_alltypes(&stream, mode))
+    {
+        printf("Parsing failed: %s\n", PB_GET_ERROR(&stream));
+        return 1;
+    } else {
+        return 0;
+    }
+}
--- a/tests/alltypes_callback/encode_alltypes_callback.c
+++ b/tests/alltypes_callback/encode_alltypes_callback.c
@@ -0,0 +1,397 @@
+/* Attempts to test all the datatypes supported by ProtoBuf when used as callback fields.
+ * Note that normally there would be no reason to use callback fields for this,
+ * because each encoder defined here only gives a single field.
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <pb_encode.h>
+#include "alltypes.pb.h"
+#include "test_helpers.h"
+
+static bool write_varint(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_varint(stream, (long)*arg);
+}
+
+static bool write_svarint(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_svarint(stream, (long)*arg);
+}
+
+static bool write_fixed32(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_fixed32(stream, *arg);
+}
+
+static bool write_fixed64(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_fixed64(stream, *arg);
+}
+
+static bool write_string(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_string(stream, *arg, strlen(*arg));
+}
+
+static bool write_submsg(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+   
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, SubMessage_fields, *arg);
+}
+
+static bool write_emptymsg(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    EmptyMessage emptymsg = {0};
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, EmptyMessage_fields, &emptymsg);
+}
+
+static bool write_repeated_varint(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_varint(stream, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_varint(stream, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_varint(stream, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_varint(stream, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_varint(stream, (long)*arg);
+}
+
+static bool write_repeated_svarint(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_svarint(stream, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_svarint(stream, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_svarint(stream, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_svarint(stream, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_svarint(stream, (long)*arg);
+}
+
+static bool write_repeated_fixed32(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    uint32_t dummy = 0;
+
+    /* Make it a packed field */
+    return pb_encode_tag(stream, PB_WT_STRING, field->tag) &&
+           pb_encode_varint(stream, 5 * 4) && /* Number of bytes */
+           pb_encode_fixed32(stream, &dummy) &&
+           pb_encode_fixed32(stream, &dummy) &&
+           pb_encode_fixed32(stream, &dummy) &&
+           pb_encode_fixed32(stream, &dummy) &&
+           pb_encode_fixed32(stream, *arg);
+}
+
+static bool write_repeated_fixed64(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    uint64_t dummy = 0;
+
+    /* Make it a packed field */
+    return pb_encode_tag(stream, PB_WT_STRING, field->tag) &&
+           pb_encode_varint(stream, 5 * 8) && /* Number of bytes */
+           pb_encode_fixed64(stream, &dummy) &&
+           pb_encode_fixed64(stream, &dummy) &&
+           pb_encode_fixed64(stream, &dummy) &&
+           pb_encode_fixed64(stream, &dummy) &&
+           pb_encode_fixed64(stream, *arg);
+}
+
+static bool write_repeated_string(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_string(stream, 0, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_string(stream, 0, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_string(stream, 0, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_string(stream, 0, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_string(stream, *arg, strlen(*arg));
+}
+
+static bool write_repeated_submsg(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    SubMessage dummy = {""};
+
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, SubMessage_fields, &dummy) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, SubMessage_fields, &dummy) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, SubMessage_fields, &dummy) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, SubMessage_fields, &dummy) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, SubMessage_fields, *arg);
+}
+
+static bool write_limits(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    Limits limits = {0};
+    limits.int32_min  = INT32_MIN;
+    limits.int32_max  = INT32_MAX;
+    limits.uint32_min = 0;
+    limits.uint32_max = UINT32_MAX;
+    limits.int64_min  = INT64_MIN;
+    limits.int64_max  = INT64_MAX;
+    limits.uint64_min = 0;
+    limits.uint64_max = UINT64_MAX;
+    limits.enum_min   = HugeEnum_Negative;
+    limits.enum_max   = HugeEnum_Positive;
+   
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, Limits_fields, &limits);
+}
+
+static bool write_repeated_emptymsg(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    EmptyMessage emptymsg = {0};
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, EmptyMessage_fields, &emptymsg) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, EmptyMessage_fields, &emptymsg) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, EmptyMessage_fields, &emptymsg) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, EmptyMessage_fields, &emptymsg) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, EmptyMessage_fields, &emptymsg);
+}
+
+int main(int argc, char **argv)
+{
+    int mode = (argc > 1) ? atoi(argv[1]) : 0;
+
+    /* Values for use from callbacks through pointers. */
+    uint32_t    req_fixed32     = 1008;
+    int32_t     req_sfixed32    = -1009;
+    float       req_float       = 1010.0f;
+    uint64_t    req_fixed64     = 1011;
+    int64_t     req_sfixed64    = -1012;
+    double      req_double      = 1013.0;
+    SubMessage  req_submsg      = {"1016", 1016};
+    
+    uint32_t    rep_fixed32     = 2008;
+    int32_t     rep_sfixed32    = -2009;
+    float       rep_float       = 2010.0f;
+    uint64_t    rep_fixed64     = 2011;
+    int64_t     rep_sfixed64    = -2012;
+    double      rep_double      = 2013.0;
+    SubMessage  rep_submsg      = {"2016", 2016, true, 2016};
+    
+    uint32_t    opt_fixed32     = 3048;
+    int32_t     opt_sfixed32    = 3049;
+    float       opt_float       = 3050.0f;
+    uint64_t    opt_fixed64     = 3051;
+    int64_t     opt_sfixed64    = 3052;
+    double      opt_double      = 3053.0f;
+    SubMessage  opt_submsg      = {"3056", 3056};
+    
+    /* Bind callbacks for required fields */
+    AllTypes alltypes = {{{0}}};
+    
+    alltypes.req_int32.funcs.encode = &write_varint;
+    alltypes.req_int32.arg = (void*)-1001;
+    
+    alltypes.req_int64.funcs.encode = &write_varint;
+    alltypes.req_int64.arg = (void*)-1002;
+    
+    alltypes.req_uint32.funcs.encode = &write_varint;
+    alltypes.req_uint32.arg = (void*)1003;
+
+    alltypes.req_uint32.funcs.encode = &write_varint;
+    alltypes.req_uint32.arg = (void*)1003;    
+    
+    alltypes.req_uint64.funcs.encode = &write_varint;
+    alltypes.req_uint64.arg = (void*)1004;
+    
+    alltypes.req_sint32.funcs.encode = &write_svarint;
+    alltypes.req_sint32.arg = (void*)-1005;   
+    
+    alltypes.req_sint64.funcs.encode = &write_svarint;
+    alltypes.req_sint64.arg = (void*)-1006;   
+    
+    alltypes.req_bool.funcs.encode = &write_varint;
+    alltypes.req_bool.arg = (void*)true;   
+    
+    alltypes.req_fixed32.funcs.encode = &write_fixed32;
+    alltypes.req_fixed32.arg = &req_fixed32;
+    
+    alltypes.req_sfixed32.funcs.encode = &write_fixed32;
+    alltypes.req_sfixed32.arg = &req_sfixed32;
+    
+    alltypes.req_float.funcs.encode = &write_fixed32;
+    alltypes.req_float.arg = &req_float;
+    
+    alltypes.req_fixed64.funcs.encode = &write_fixed64;
+    alltypes.req_fixed64.arg = &req_fixed64;
+    
+    alltypes.req_sfixed64.funcs.encode = &write_fixed64;
+    alltypes.req_sfixed64.arg = &req_sfixed64;
+    
+    alltypes.req_double.funcs.encode = &write_fixed64;
+    alltypes.req_double.arg = &req_double;
+    
+    alltypes.req_string.funcs.encode = &write_string;
+    alltypes.req_string.arg = "1014";
+    
+    alltypes.req_bytes.funcs.encode = &write_string;
+    alltypes.req_bytes.arg = "1015";
+    
+    alltypes.req_submsg.funcs.encode = &write_submsg;
+    alltypes.req_submsg.arg = &req_submsg;
+    
+    alltypes.req_enum.funcs.encode = &write_varint;
+    alltypes.req_enum.arg = (void*)MyEnum_Truth;
+    
+    alltypes.req_emptymsg.funcs.encode = &write_emptymsg;
+    
+    /* Bind callbacks for repeated fields */
+    alltypes.rep_int32.funcs.encode = &write_repeated_varint;
+    alltypes.rep_int32.arg = (void*)-2001;
+    
+    alltypes.rep_int64.funcs.encode = &write_repeated_varint;
+    alltypes.rep_int64.arg = (void*)-2002;
+    
+    alltypes.rep_uint32.funcs.encode = &write_repeated_varint;
+    alltypes.rep_uint32.arg = (void*)2003;
+    
+    alltypes.rep_uint64.funcs.encode = &write_repeated_varint;
+    alltypes.rep_uint64.arg = (void*)2004;
+    
+    alltypes.rep_sint32.funcs.encode = &write_repeated_svarint;
+    alltypes.rep_sint32.arg = (void*)-2005;
+    
+    alltypes.rep_sint64.funcs.encode = &write_repeated_svarint;
+    alltypes.rep_sint64.arg = (void*)-2006;
+    
+    alltypes.rep_bool.funcs.encode = &write_repeated_varint;
+    alltypes.rep_bool.arg = (void*)true;
+    
+    alltypes.rep_fixed32.funcs.encode = &write_repeated_fixed32;
+    alltypes.rep_fixed32.arg = &rep_fixed32;
+
+    alltypes.rep_sfixed32.funcs.encode = &write_repeated_fixed32;
+    alltypes.rep_sfixed32.arg = &rep_sfixed32;
+    
+    alltypes.rep_float.funcs.encode = &write_repeated_fixed32;
+    alltypes.rep_float.arg = &rep_float;
+    
+    alltypes.rep_fixed64.funcs.encode = &write_repeated_fixed64;
+    alltypes.rep_fixed64.arg = &rep_fixed64;
+
+    alltypes.rep_sfixed64.funcs.encode = &write_repeated_fixed64;
+    alltypes.rep_sfixed64.arg = &rep_sfixed64;
+    
+    alltypes.rep_double.funcs.encode = &write_repeated_fixed64;
+    alltypes.rep_double.arg = &rep_double;
+    
+    alltypes.rep_string.funcs.encode = &write_repeated_string;
+    alltypes.rep_string.arg = "2014";
+    
+    alltypes.rep_bytes.funcs.encode = &write_repeated_string;
+    alltypes.rep_bytes.arg = "2015";
+    
+    alltypes.rep_submsg.funcs.encode = &write_repeated_submsg;
+    alltypes.rep_submsg.arg = &rep_submsg;
+    
+    alltypes.rep_enum.funcs.encode = &write_repeated_varint;
+    alltypes.rep_enum.arg = (void*)MyEnum_Truth;
+    
+    alltypes.rep_emptymsg.funcs.encode = &write_repeated_emptymsg;
+    
+    alltypes.req_limits.funcs.encode = &write_limits;
+    
+    /* Bind callbacks for optional fields */
+    if (mode != 0)
+    {
+        alltypes.opt_int32.funcs.encode = &write_varint;
+        alltypes.opt_int32.arg = (void*)3041;
+        
+        alltypes.opt_int64.funcs.encode = &write_varint;
+        alltypes.opt_int64.arg = (void*)3042;
+        
+        alltypes.opt_uint32.funcs.encode = &write_varint;
+        alltypes.opt_uint32.arg = (void*)3043;
+        
+        alltypes.opt_uint64.funcs.encode = &write_varint;
+        alltypes.opt_uint64.arg = (void*)3044;
+        
+        alltypes.opt_sint32.funcs.encode = &write_svarint;
+        alltypes.opt_sint32.arg = (void*)3045;
+        
+        alltypes.opt_sint64.funcs.encode = &write_svarint;
+        alltypes.opt_sint64.arg = (void*)3046;
+        
+        alltypes.opt_bool.funcs.encode = &write_varint;
+        alltypes.opt_bool.arg = (void*)true;
+
+        alltypes.opt_fixed32.funcs.encode = &write_fixed32;
+        alltypes.opt_fixed32.arg = &opt_fixed32;
+        
+        alltypes.opt_sfixed32.funcs.encode = &write_fixed32;
+        alltypes.opt_sfixed32.arg = &opt_sfixed32;
+        
+        alltypes.opt_float.funcs.encode = &write_fixed32;
+        alltypes.opt_float.arg = &opt_float;
+        
+        alltypes.opt_fixed64.funcs.encode = &write_fixed64;
+        alltypes.opt_fixed64.arg = &opt_fixed64;
+        
+        alltypes.opt_sfixed64.funcs.encode = &write_fixed64;
+        alltypes.opt_sfixed64.arg = &opt_sfixed64;
+        
+        alltypes.opt_double.funcs.encode = &write_fixed64;
+        alltypes.opt_double.arg = &opt_double;
+        
+        alltypes.opt_string.funcs.encode = &write_string;
+        alltypes.opt_string.arg = "3054";
+        
+        alltypes.opt_bytes.funcs.encode = &write_string;
+        alltypes.opt_bytes.arg = "3055";
+        
+        alltypes.opt_submsg.funcs.encode = &write_submsg;
+        alltypes.opt_submsg.arg = &opt_submsg;
+        
+        alltypes.opt_enum.funcs.encode = &write_varint;
+        alltypes.opt_enum.arg = (void*)MyEnum_Truth;
+        
+        alltypes.opt_emptymsg.funcs.encode = &write_emptymsg;
+    }
+    
+    alltypes.end.funcs.encode = &write_varint;
+    alltypes.end.arg = (void*)1099;
+    
+    {
+        uint8_t buffer[2048];
+        pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
+        
+        /* Now encode it and check if we succeeded. */
+        if (pb_encode(&stream, AllTypes_fields, &alltypes))
+        {
+            SET_BINARY_MODE(stdout);
+            fwrite(buffer, 1, stream.bytes_written, stdout);
+            return 0; /* Success */
+        }
+        else
+        {
+            fprintf(stderr, "Encoding failed: %s\n", PB_GET_ERROR(&stream));
+            return 1; /* Failure */
+        }
+    }
+}
--- a/tests/alltypes_pointer/SConscript
+++ b/tests/alltypes_pointer/SConscript
@@ -1,23 +1,38 @@
 # Encode the AllTypes message using pointers for all fields, and verify the
 # output against the normal AllTypes test case.

-Import("env")
+Import("env", "malloc_env")

 c = Copy("$TARGET", "$SOURCE")
 env.Command("alltypes.proto", "#alltypes/alltypes.proto", c)

 env.NanopbProto(["alltypes", "alltypes.options"])
-enc = env.Program(["encode_alltypes_pointer.c", "alltypes.pb.c", "$COMMON/pb_encode.o"])
+enc = malloc_env.Program(["encode_alltypes_pointer.c",
+                          "alltypes.pb.c",
+                          "$COMMON/pb_encode_with_malloc.o",
+                          "$COMMON/malloc_wrappers.o"])
+dec = malloc_env.Program(["decode_alltypes_pointer.c",
+                          "alltypes.pb.c",
+                          "$COMMON/pb_decode_with_malloc.o",
+                          "$COMMON/malloc_wrappers.o"])

-refdec = "$BUILD/alltypes/decode_alltypes$PROGSUFFIX"
-
-# Encode and compare results
+# Encode and compare results to non-pointer alltypes test case
 env.RunTest(enc)
-env.RunTest("decode_alltypes.output", [refdec, "encode_alltypes_pointer.output"])
 env.Compare(["encode_alltypes_pointer.output", "$BUILD/alltypes/encode_alltypes.output"])

+# Decode (under valgrind if available)
+valgrind = env.WhereIs('valgrind')
+kwargs = {}
+if valgrind:
+    kwargs['COMMAND'] = valgrind
+    kwargs['ARGS'] = ["-q", dec[0].abspath]
+
+env.RunTest("decode_alltypes.output", [dec, "encode_alltypes_pointer.output"], **kwargs)
+
 # Do the same thing with the optional fields present
 env.RunTest("optionals.output", enc, ARGS = ['1'])
-env.RunTest("optionals.decout", [refdec, "optionals.output"], ARGS = ['1'])
 env.Compare(["optionals.output", "$BUILD/alltypes/optionals.output"])

+kwargs['ARGS'] = kwargs.get('ARGS', []) + ['1']
+env.RunTest("optionals.decout", [dec, "optionals.output"], **kwargs)
+
--- a/tests/alltypes_pointer/decode_alltypes_pointer.c
+++ b/tests/alltypes_pointer/decode_alltypes_pointer.c
@@ -0,0 +1,174 @@
+#include <stdio.h>
+#include <string.h>
+#include <stdlib.h>
+#include <pb_decode.h>
+#include "alltypes.pb.h"
+#include "test_helpers.h"
+
+#define TEST(x) if (!(x)) { \
+    fprintf(stderr, "Test " #x " failed.\n"); \
+    status = false; \
+    }
+
+/* This function is called once from main(), it handles
+   the decoding and checks the fields. */
+bool check_alltypes(pb_istream_t *stream, int mode)
+{
+    bool status = true;
+    AllTypes alltypes;
+    
+    /* Fill with garbage to better detect initialization errors */
+    memset(&alltypes, 0xAA, sizeof(alltypes));
+    alltypes.extensions = 0;
+    
+    if (!pb_decode(stream, AllTypes_fields, &alltypes))
+        return false;
+    
+    TEST(alltypes.req_int32     && *alltypes.req_int32         == -1001);
+    TEST(alltypes.req_int64     && *alltypes.req_int64         == -1002);
+    TEST(alltypes.req_uint32    && *alltypes.req_uint32        == 1003);
+    TEST(alltypes.req_uint64    && *alltypes.req_uint64        == 1004);
+    TEST(alltypes.req_sint32    && *alltypes.req_sint32        == -1005);
+    TEST(alltypes.req_sint64    && *alltypes.req_sint64        == -1006);
+    TEST(alltypes.req_bool      && *alltypes.req_bool          == true);
+    
+    TEST(alltypes.req_fixed32   && *alltypes.req_fixed32       == 1008);
+    TEST(alltypes.req_sfixed32  && *alltypes.req_sfixed32      == -1009);
+    TEST(alltypes.req_float     && *alltypes.req_float         == 1010.0f);
+    
+    TEST(alltypes.req_fixed64   && *alltypes.req_fixed64       == 1011);
+    TEST(alltypes.req_sfixed64  && *alltypes.req_sfixed64      == -1012);
+    TEST(alltypes.req_double    && *alltypes.req_double        == 1013.0f);
+    
+    TEST(alltypes.req_string    && strcmp(alltypes.req_string, "1014") == 0);
+    TEST(alltypes.req_bytes     && alltypes.req_bytes->size == 4);
+    TEST(alltypes.req_bytes     && memcmp(&alltypes.req_bytes->bytes, "1015", 4) == 0);
+    TEST(alltypes.req_submsg    && alltypes.req_submsg->substuff1
+                                && strcmp(alltypes.req_submsg->substuff1, "1016") == 0);
+    TEST(alltypes.req_submsg    && alltypes.req_submsg->substuff2
+                                && *alltypes.req_submsg->substuff2 == 1016);
+    TEST(*alltypes.req_enum == MyEnum_Truth);
+
+    TEST(alltypes.rep_int32_count == 5 && alltypes.rep_int32[4] == -2001 && alltypes.rep_int32[0] == 0);
+    TEST(alltypes.rep_int64_count == 5 && alltypes.rep_int64[4] == -2002 && alltypes.rep_int64[0] == 0);
+    TEST(alltypes.rep_uint32_count == 5 && alltypes.rep_uint32[4] == 2003 && alltypes.rep_uint32[0] == 0);
+    TEST(alltypes.rep_uint64_count == 5 && alltypes.rep_uint64[4] == 2004 && alltypes.rep_uint64[0] == 0);
+    TEST(alltypes.rep_sint32_count == 5 && alltypes.rep_sint32[4] == -2005 && alltypes.rep_sint32[0] == 0);
+    TEST(alltypes.rep_sint64_count == 5 && alltypes.rep_sint64[4] == -2006 && alltypes.rep_sint64[0] == 0);
+    TEST(alltypes.rep_bool_count == 5 && alltypes.rep_bool[4] == true && alltypes.rep_bool[0] == false);
+    
+    TEST(alltypes.rep_fixed32_count == 5 && alltypes.rep_fixed32[4] == 2008 && alltypes.rep_fixed32[0] == 0);
+    TEST(alltypes.rep_sfixed32_count == 5 && alltypes.rep_sfixed32[4] == -2009 && alltypes.rep_sfixed32[0] == 0);
+    TEST(alltypes.rep_float_count == 5 && alltypes.rep_float[4] == 2010.0f && alltypes.rep_float[0] == 0.0f);
+    
+    TEST(alltypes.rep_fixed64_count == 5 && alltypes.rep_fixed64[4] == 2011 && alltypes.rep_fixed64[0] == 0);
+    TEST(alltypes.rep_sfixed64_count == 5 && alltypes.rep_sfixed64[4] == -2012 && alltypes.rep_sfixed64[0] == 0);
+    TEST(alltypes.rep_double_count == 5 && alltypes.rep_double[4] == 2013.0 && alltypes.rep_double[0] == 0.0);
+    
+    TEST(alltypes.rep_string_count == 5 && strcmp(alltypes.rep_string[4], "2014") == 0 && alltypes.rep_string[0][0] == '\0');
+    TEST(alltypes.rep_bytes_count == 5 && alltypes.rep_bytes[4]->size == 4 && alltypes.rep_bytes[0]->size == 0);
+    TEST(memcmp(&alltypes.rep_bytes[4]->bytes, "2015", 4) == 0);
+
+    TEST(alltypes.rep_submsg_count == 5);
+    TEST(strcmp(alltypes.rep_submsg[4].substuff1, "2016") == 0 && alltypes.rep_submsg[0].substuff1[0] == '\0');
+    TEST(*alltypes.rep_submsg[4].substuff2 == 2016 && *alltypes.rep_submsg[0].substuff2 == 0);
+    TEST(*alltypes.rep_submsg[4].substuff3 == 2016 && alltypes.rep_submsg[0].substuff3 == NULL);
+    
+    TEST(alltypes.rep_enum_count == 5 && alltypes.rep_enum[4] == MyEnum_Truth && alltypes.rep_enum[0] == MyEnum_Zero);
+    TEST(alltypes.rep_emptymsg_count == 5);
+
+    if (mode == 0)
+    {
+        /* Expect that optional values are not present */
+        TEST(alltypes.opt_int32         == NULL);
+        TEST(alltypes.opt_int64         == NULL);
+        TEST(alltypes.opt_uint32        == NULL);
+        TEST(alltypes.opt_uint64        == NULL);
+        TEST(alltypes.opt_sint32        == NULL);
+        TEST(alltypes.opt_sint64        == NULL);
+        TEST(alltypes.opt_bool          == NULL);
+        
+        TEST(alltypes.opt_fixed32       == NULL);
+        TEST(alltypes.opt_sfixed32      == NULL);
+        TEST(alltypes.opt_float         == NULL);
+        TEST(alltypes.opt_fixed64       == NULL);
+        TEST(alltypes.opt_sfixed64      == NULL);
+        TEST(alltypes.opt_double        == NULL);
+        
+        TEST(alltypes.opt_string        == NULL);
+        TEST(alltypes.opt_bytes         == NULL);
+        TEST(alltypes.opt_submsg        == NULL);
+        TEST(alltypes.opt_enum          == NULL);
+    }
+    else
+    {
+        /* Expect filled-in values */
+        TEST(alltypes.opt_int32 && *alltypes.opt_int32      == 3041);
+        TEST(alltypes.opt_int64 && *alltypes.opt_int64      == 3042);
+        TEST(alltypes.opt_uint32 && *alltypes.opt_uint32    == 3043);
+        TEST(alltypes.opt_uint64 && *alltypes.opt_uint64    == 3044);
+        TEST(alltypes.opt_sint32 && *alltypes.opt_sint32    == 3045);
+        TEST(alltypes.opt_sint64 && *alltypes.opt_sint64    == 3046);
+        TEST(alltypes.opt_bool && *alltypes.opt_bool        == true);
+        
+        TEST(alltypes.opt_fixed32 && *alltypes.opt_fixed32  == 3048);
+        TEST(alltypes.opt_sfixed32 && *alltypes.opt_sfixed32== 3049);
+        TEST(alltypes.opt_float && *alltypes.opt_float      == 3050.0f);
+        TEST(alltypes.opt_fixed64 && *alltypes.opt_fixed64  == 3051);
+        TEST(alltypes.opt_sfixed64 && *alltypes.opt_sfixed64== 3052);
+        TEST(alltypes.opt_double && *alltypes.opt_double    == 3053.0);
+        
+        TEST(alltypes.opt_string && strcmp(alltypes.opt_string, "3054") == 0);
+        TEST(alltypes.opt_bytes && alltypes.opt_bytes->size == 4);
+        TEST(alltypes.opt_bytes && memcmp(&alltypes.opt_bytes->bytes, "3055", 4) == 0);
+        TEST(alltypes.opt_submsg && strcmp(alltypes.opt_submsg->substuff1, "3056") == 0);
+        TEST(alltypes.opt_submsg && *alltypes.opt_submsg->substuff2 == 3056);
+        TEST(alltypes.opt_enum && *alltypes.opt_enum == MyEnum_Truth);
+        TEST(alltypes.opt_emptymsg);
+    }
+    
+    TEST(alltypes.req_limits->int32_min && *alltypes.req_limits->int32_min   == INT32_MIN);
+    TEST(alltypes.req_limits->int32_max && *alltypes.req_limits->int32_max   == INT32_MAX);
+    TEST(alltypes.req_limits->uint32_min && *alltypes.req_limits->uint32_min == 0);
+    TEST(alltypes.req_limits->uint32_max && *alltypes.req_limits->uint32_max == UINT32_MAX);
+    TEST(alltypes.req_limits->int64_min && *alltypes.req_limits->int64_min   == INT64_MIN);
+    TEST(alltypes.req_limits->int64_max && *alltypes.req_limits->int64_max   == INT64_MAX);
+    TEST(alltypes.req_limits->uint64_min && *alltypes.req_limits->uint64_min == 0);
+    TEST(alltypes.req_limits->uint64_max && *alltypes.req_limits->uint64_max == UINT64_MAX);
+    TEST(alltypes.req_limits->enum_min && *alltypes.req_limits->enum_min     == HugeEnum_Negative);
+    TEST(alltypes.req_limits->enum_max && *alltypes.req_limits->enum_max     == HugeEnum_Positive);
+    
+    TEST(alltypes.end && *alltypes.end == 1099);
+
+    pb_release(AllTypes_fields, &alltypes);
+
+    return status;
+}
+
+int main(int argc, char **argv)
+{
+    uint8_t buffer[1024];
+    size_t count;
+    pb_istream_t stream;
+    
+    /* Whether to expect the optional values or the default values. */
+    int mode = (argc > 1) ? atoi(argv[1]) : 0;
+    
+    /* Read the data into buffer */
+    SET_BINARY_MODE(stdin);
+    count = fread(buffer, 1, sizeof(buffer), stdin);
+    
+    /* Construct a pb_istream_t for reading from the buffer */
+    stream = pb_istream_from_buffer(buffer, count);
+    
+    /* Decode and verify the message */
+    if (!check_alltypes(&stream, mode))
+    {
+        fprintf(stderr, "Test failed: %s\n", PB_GET_ERROR(&stream));
+        return 1;
+    }
+    else
+    {
+        return 0;
+    }
+}
--- a/tests/alltypes_pointer/encode_alltypes_pointer.c
+++ b/tests/alltypes_pointer/encode_alltypes_pointer.c
@@ -27,7 +27,7 @@ int main(int argc, char **argv)
    int64_t     req_sfixed64      = -1012;
    double      req_double        = 1013.0;
    char*       req_string        = "1014";
-    pb_bytes_ptr_t req_bytes      = {4, (uint8_t*)"1015"};
+    PB_BYTES_ARRAY_T(4) req_bytes = {4, {'1', '0', '1', '5'}};
    static int32_t req_substuff   = 1016;
    SubMessage  req_submsg        = {"1016", &req_substuff};
    MyEnum      req_enum          = MyEnum_Truth;
@@ -50,7 +50,8 @@ int main(int argc, char **argv)
    int64_t     rep_sfixed64[5]   = {0, 0, 0, 0, -2012};
    double      rep_double[5]     = {0, 0, 0, 0, 2013.0f};
    char*       rep_string[5]     = {"", "", "", "", "2014"};
-    pb_bytes_ptr_t rep_bytes[5]   = {{0,0}, {0,0}, {0,0}, {0,0}, {4, (uint8_t*)"2015"}};
+    static PB_BYTES_ARRAY_T(4) rep_bytes_4 = {4, {'2', '0', '1', '5'}};
+    pb_bytes_array_t *rep_bytes[5]= {NULL, NULL, NULL, NULL, (pb_bytes_array_t*)&rep_bytes_4};
    static int32_t rep_sub2zero   = 0;
    static int32_t rep_substuff2  = 2016;
    static uint32_t rep_substuff3 = 2016;
@@ -77,7 +78,7 @@ int main(int argc, char **argv)
    int64_t     opt_sfixed64      = 3052;
    double      opt_double        = 3053.0;
    char*       opt_string        = "3054";
-    pb_bytes_ptr_t opt_bytes      = {4, (uint8_t*)"3055"};
+    PB_BYTES_ARRAY_T(4) opt_bytes = {4, {'3', '0', '5', '5'}};
    static int32_t opt_substuff   = 3056;
    SubMessage  opt_submsg        = {"3056", &opt_substuff};
    MyEnum      opt_enum          = MyEnum_Truth;
@@ -117,7 +118,7 @@ int main(int argc, char **argv)
    alltypes.req_sfixed64      = &req_sfixed64;
    alltypes.req_double        = &req_double;
    alltypes.req_string        = req_string;
-    alltypes.req_bytes         = &req_bytes;
+    alltypes.req_bytes         = (pb_bytes_array_t*)&req_bytes;
    alltypes.req_submsg        = &req_submsg;
    alltypes.req_enum          = &req_enum;
    alltypes.req_emptymsg      = &req_emptymsg;
@@ -159,7 +160,7 @@ int main(int argc, char **argv)
        alltypes.opt_sfixed64      = &opt_sfixed64;
        alltypes.opt_double        = &opt_double;
        alltypes.opt_string        = opt_string;
-        alltypes.opt_bytes         = &opt_bytes;
+        alltypes.opt_bytes         = (pb_bytes_array_t*)&opt_bytes;
        alltypes.opt_submsg        = &opt_submsg;
        alltypes.opt_enum          = &opt_enum;
        alltypes.opt_emptymsg      = &opt_emptymsg;
--- a/tests/basic_buffer/decode_buffer.c
+++ b/tests/basic_buffer/decode_buffer.c
@@ -16,7 +16,7 @@
 bool print_person(pb_istream_t *stream)
 {
    int i;
-    Person person;
+    Person person = Person_init_zero;
    
    if (!pb_decode(stream, Person_fields, &person))
        return false;
--- a/tests/basic_stream/decode_stream.c
+++ b/tests/basic_stream/decode_stream.c
@@ -12,7 +12,7 @@
 bool print_person(pb_istream_t *stream)
 {
    int i;
-    Person person;
+    Person person = Person_init_zero;
    
    if (!pb_decode(stream, Person_fields, &person))
        return false;
--- a/tests/buffer_only/SConscript
+++ b/tests/buffer_only/SConscript
@@ -4,8 +4,6 @@ Import("env")

 # Take copy of the files for custom build.
 c = Copy("$TARGET", "$SOURCE")
-env.Command("pb_encode.c", "#../pb_encode.c", c)
-env.Command("pb_decode.c", "#../pb_decode.c", c)
 env.Command("alltypes.pb.h", "$BUILD/alltypes/alltypes.pb.h", c)
 env.Command("alltypes.pb.c", "$BUILD/alltypes/alltypes.pb.c", c)
 env.Command("encode_alltypes.c", "$BUILD/alltypes/encode_alltypes.c", c)
@@ -15,9 +13,15 @@ env.Command("decode_alltypes.c", "$BUILD/alltypes/decode_alltypes.c", c)
 opts = env.Clone()
 opts.Append(CPPDEFINES = {'PB_BUFFER_ONLY': 1})

+# Build new version of core
+strict = opts.Clone()
+strict.Append(CFLAGS = strict['CORECFLAGS'])
+strict.Object("pb_decode_bufonly.o", "$NANOPB/pb_decode.c")
+strict.Object("pb_encode_bufonly.o", "$NANOPB/pb_encode.c")
+
 # Now build and run the test normally.
-enc = opts.Program(["encode_alltypes.c", "alltypes.pb.c", "pb_encode.c"])
-dec = opts.Program(["decode_alltypes.c", "alltypes.pb.c", "pb_decode.c"])
+enc = opts.Program(["encode_alltypes.c", "alltypes.pb.c", "pb_encode_bufonly.o"])
+dec = opts.Program(["decode_alltypes.c", "alltypes.pb.c", "pb_decode_bufonly.o"])

 env.RunTest(enc)
 env.RunTest([dec, "encode_alltypes.output"])
--- a/tests/common/SConscript
+++ b/tests/common/SConscript
@@ -8,6 +8,7 @@ env.NanopbProto("unittestproto")
 # Protocol definitions for basic_buffer/stream tests
 env.NanopbProto("person")

+#--------------------------------------------
 # Binaries of the pb_decode.c and pb_encode.c
 # These are built using more strict warning flags.
 strict = env.Clone()
@@ -15,3 +16,31 @@ strict.Append(CFLAGS = strict['CORECFLAGS'])
 strict.Object("pb_decode.o", "$NANOPB/pb_decode.c")
 strict.Object("pb_encode.o", "$NANOPB/pb_encode.c")

+#-----------------------------------------------
+# Binaries of pb_decode etc. with malloc support
+# Uses malloc_wrappers.c to count allocations.
+malloc_env = env.Clone()
+malloc_env.Append(CPPDEFINES = {'PB_ENABLE_MALLOC': 1,
+                                'PB_SYSTEM_HEADER': '\\"malloc_wrappers_syshdr.h\\"'})
+malloc_env.Append(CPPPATH = ["$COMMON"])
+
+if 'SYSHDR' in malloc_env:
+    malloc_env.Append(CPPDEFINES = {'PB_OLD_SYSHDR': malloc_env['SYSHDR']})
+
+# Disable libmudflap, because it will confuse valgrind
+# and other memory leak detection tools.
+if '-fmudflap' in env["CCFLAGS"]:
+    malloc_env["CCFLAGS"].remove("-fmudflap")
+    malloc_env["LINKFLAGS"].remove("-fmudflap")
+    malloc_env["LIBS"].remove("mudflap")
+
+malloc_strict = malloc_env.Clone()
+malloc_strict.Append(CFLAGS = malloc_strict['CORECFLAGS'])
+malloc_strict.Object("pb_decode_with_malloc.o", "$NANOPB/pb_decode.c")
+malloc_strict.Object("pb_encode_with_malloc.o", "$NANOPB/pb_encode.c")
+
+malloc_env.Object("malloc_wrappers.o", "malloc_wrappers.c")
+malloc_env.Depends("$NANOPB/pb.h", ["malloc_wrappers_syshdr.h", "malloc_wrappers.h"])
+
+Export("malloc_env")
+
--- a/tests/common/malloc_wrappers.c
+++ b/tests/common/malloc_wrappers.c
@@ -0,0 +1,54 @@
+#include "malloc_wrappers.h"
+#include <stdint.h>
+#include <assert.h>
+#include <string.h>
+
+static size_t alloc_count = 0;
+
+/* Allocate memory and place check values before and after. */
+void* malloc_with_check(size_t size)
+{
+    size_t size32 = (size + 3) / 4 + 3;
+    uint32_t *buf = malloc(size32 * sizeof(uint32_t));
+    buf[0] = size32;
+    buf[1] = 0xDEADBEEF;
+    buf[size32 - 1] = 0xBADBAD;
+    return buf + 2;
+}
+
+/* Free memory allocated with malloc_with_check() and do the checks. */
+void free_with_check(void *mem)
+{
+    uint32_t *buf = (uint32_t*)mem - 2;
+    assert(buf[1] == 0xDEADBEEF);
+    assert(buf[buf[0] - 1] == 0xBADBAD);
+    free(buf);
+}
+
+/* Track memory usage */
+void* counting_realloc(void *ptr, size_t size)
+{
+    /* Don't allocate crazy amounts of RAM when fuzzing */
+    if (size > 1000000)
+        return NULL;
+
+    if (!ptr && size)
+        alloc_count++;
+    
+    return realloc(ptr, size);
+}
+
+void counting_free(void *ptr)
+{
+    if (ptr)
+    {
+        assert(alloc_count > 0);
+        alloc_count--;
+        free(ptr);
+    }
+}
+
+size_t get_alloc_count()
+{
+    return alloc_count;
+}
--- a/tests/common/malloc_wrappers.h
+++ b/tests/common/malloc_wrappers.h
@@ -0,0 +1,7 @@
+#include <stdlib.h>
+
+void* malloc_with_check(size_t size);
+void free_with_check(void *mem);
+void* counting_realloc(void *ptr, size_t size);
+void counting_free(void *ptr);
+size_t get_alloc_count();
--- a/tests/common/malloc_wrappers_syshdr.h
+++ b/tests/common/malloc_wrappers_syshdr.h
@@ -0,0 +1,15 @@
+/* This is just a wrapper in order to get our own malloc wrappers into nanopb core. */
+
+#define pb_realloc(ptr,size) counting_realloc(ptr,size)
+#define pb_free(ptr) counting_free(ptr)
+
+#ifdef PB_OLD_SYSHDR
+#include PB_OLD_SYSHDR
+#else
+#include <stdint.h>
+#include <stddef.h>
+#include <stdbool.h>
+#include <string.h>
+#endif
+
+#include <malloc_wrappers.h>
--- a/tests/decode_unittests/decode_unittests.c
+++ b/tests/decode_unittests/decode_unittests.c
@@ -1,4 +1,5 @@
 /* This includes the whole .c file to get access to static functions. */
+#define PB_ENABLE_MALLOC
 #include "pb_decode.c"

 #include <stdio.h>
@@ -299,6 +300,28 @@ int main()
              dest.submsg.data_count == 5)
    }
    
+    {
+        pb_istream_t s = {0};
+        void *data = NULL;
+        
+        COMMENT("Testing allocate_field")
+        TEST(allocate_field(&s, &data, 10, 10) && data != NULL);
+        TEST(allocate_field(&s, &data, 10, 20) && data != NULL);
+        
+        {
+            void *oldvalue = data;
+            size_t very_big = (size_t)-1;
+            size_t somewhat_big = very_big / 2 + 1;
+            size_t not_so_big = (size_t)1 << (4 * sizeof(size_t));
+        
+            TEST(!allocate_field(&s, &data, very_big, 2) && data == oldvalue);
+            TEST(!allocate_field(&s, &data, somewhat_big, 2) && data == oldvalue);
+            TEST(!allocate_field(&s, &data, not_so_big, not_so_big) && data == oldvalue);
+        }
+        
+        pb_free(data);
+    }
+    
    if (status != 0)
        fprintf(stdout, "\n\nSome tests FAILED!\n");
    
--- a/tests/encode_unittests/encode_unittests.c
+++ b/tests/encode_unittests/encode_unittests.c
@@ -282,6 +282,15 @@ int main()
                    "\x09\x0A\x07\x0A\x05\x01\x02\x03\x04\x05"))
    }

+    {
+        IntegerContainer msg = {{5, {1,2,3,4,5}}};
+        size_t size;
+        
+        COMMENT("Test pb_get_encoded_size.")
+        TEST(pb_get_encoded_size(&size, IntegerContainer_fields, &msg) &&
+             size == 9);
+    }
+    
    {
        uint8_t buffer[10];
        pb_ostream_t s;
--- a/tests/extensions/decode_extensions.c
+++ b/tests/extensions/decode_extensions.c
@@ -49,7 +49,9 @@ int main(int argc, char **argv)
    }

    /* Check that the extensions decoded properly */
+    TEST(ext1.found)
    TEST(extensionfield1 == 12345)
+    TEST(ext2.found)
    TEST(strcmp(extensionfield2.test1, "test") == 0)
    TEST(extensionfield2.test2 == 54321)
    
--- a/tests/extensions/extensions.proto
+++ b/tests/extensions/extensions.proto
@@ -1,13 +1,12 @@
 import 'alltypes.proto';

 extend AllTypes {
-    optional int32 AllTypes_extensionfield1 = 255;
+    optional int32 AllTypes_extensionfield1 = 255 [default = 5];
 }

 message ExtensionMessage {
    extend AllTypes {
        optional ExtensionMessage AllTypes_extensionfield2 = 254;
-	required ExtensionMessage AllTypes_extensionfield3 = 253;
 	repeated ExtensionMessage AllTypes_extensionfield4 = 252;
    }
    
--- a/tests/field_size_16/SConscript
+++ b/tests/field_size_16/SConscript
@@ -5,8 +5,6 @@ Import("env")

 # Take copy of the files for custom build.
 c = Copy("$TARGET", "$SOURCE")
-env.Command("pb_encode.c", "#../pb_encode.c", c)
-env.Command("pb_decode.c", "#../pb_decode.c", c)
 env.Command("encode_alltypes.c", "$BUILD/alltypes/encode_alltypes.c", c)
 env.Command("decode_alltypes.c", "$BUILD/alltypes/decode_alltypes.c", c)

@@ -16,9 +14,15 @@ env.NanopbProto(["alltypes", "alltypes.options"])
 opts = env.Clone()
 opts.Append(CPPDEFINES = {'PB_FIELD_16BIT': 1})

+# Build new version of core
+strict = opts.Clone()
+strict.Append(CFLAGS = strict['CORECFLAGS'])
+strict.Object("pb_decode_fields16.o", "$NANOPB/pb_decode.c")
+strict.Object("pb_encode_fields16.o", "$NANOPB/pb_encode.c")
+
 # Now build and run the test normally.
-enc = opts.Program(["encode_alltypes.c", "alltypes.pb.c", "pb_encode.c"])
-dec = opts.Program(["decode_alltypes.c", "alltypes.pb.c", "pb_decode.c"])
+enc = opts.Program(["encode_alltypes.c", "alltypes.pb.c", "pb_encode_fields16.o"])
+dec = opts.Program(["decode_alltypes.c", "alltypes.pb.c", "pb_decode_fields16.o"])

 env.RunTest(enc)
 env.RunTest([dec, "encode_alltypes.output"])
--- a/tests/field_size_16/alltypes.proto
+++ b/tests/field_size_16/alltypes.proto
@@ -107,5 +107,7 @@ message AllTypes {
    // Just to make sure that the size of the fields has been calculated
    // properly, i.e. otherwise a bug in last field might not be detected.
    required int32      end = 10099;
+
+    extensions 200 to 255;
 }

--- a/tests/field_size_32/SConscript
+++ b/tests/field_size_32/SConscript
@@ -5,8 +5,6 @@ Import("env")

 # Take copy of the files for custom build.
 c = Copy("$TARGET", "$SOURCE")
-env.Command("pb_encode.c", "#../pb_encode.c", c)
-env.Command("pb_decode.c", "#../pb_decode.c", c)
 env.Command("encode_alltypes.c", "$BUILD/alltypes/encode_alltypes.c", c)
 env.Command("decode_alltypes.c", "$BUILD/alltypes/decode_alltypes.c", c)

@@ -16,9 +14,15 @@ env.NanopbProto(["alltypes", "alltypes.options"])
 opts = env.Clone()
 opts.Append(CPPDEFINES = {'PB_FIELD_32BIT': 1})

+# Build new version of core
+strict = opts.Clone()
+strict.Append(CFLAGS = strict['CORECFLAGS'])
+strict.Object("pb_decode_fields32.o", "$NANOPB/pb_decode.c")
+strict.Object("pb_encode_fields32.o", "$NANOPB/pb_encode.c")
+
 # Now build and run the test normally.
-enc = opts.Program(["encode_alltypes.c", "alltypes.pb.c", "pb_encode.c"])
-dec = opts.Program(["decode_alltypes.c", "alltypes.pb.c", "pb_decode.c"])
+enc = opts.Program(["encode_alltypes.c", "alltypes.pb.c", "pb_encode_fields32.o"])
+dec = opts.Program(["decode_alltypes.c", "alltypes.pb.c", "pb_decode_fields32.o"])

 env.RunTest(enc)
 env.RunTest([dec, "encode_alltypes.output"])
--- a/tests/field_size_32/alltypes.proto
+++ b/tests/field_size_32/alltypes.proto
@@ -107,5 +107,7 @@ message AllTypes {
    // Just to make sure that the size of the fields has been calculated
    // properly, i.e. otherwise a bug in last field might not be detected.
    required int32      end = 13432099;
+
+    extensions 200 to 255;
 }

--- a/tests/fuzztest/SConscript
+++ b/tests/fuzztest/SConscript
@@ -0,0 +1,34 @@
+# Run a fuzz test to verify robustness against corrupted/malicious data.
+
+Import("env", "malloc_env")
+
+# We want both pointer and static versions of the AllTypes message
+# Prefix them with package name.
+env.Command("alltypes_static.proto", "#alltypes/alltypes.proto",
+            lambda target, source, env:
+                open(str(target[0]), 'w').write("package alltypes_static;\n"
+                                                + open(str(source[0])).read()))
+env.Command("alltypes_pointer.proto", "#alltypes/alltypes.proto",
+            lambda target, source, env:
+                open(str(target[0]), 'w').write("package alltypes_pointer;\n"
+                                                + open(str(source[0])).read()))
+
+p1 = env.NanopbProto(["alltypes_pointer", "alltypes_pointer.options"])
+p2 = env.NanopbProto(["alltypes_static", "alltypes_static.options"])
+fuzz = malloc_env.Program(["fuzztest.c",
+                    "alltypes_pointer.pb.c",
+                    "alltypes_static.pb.c",
+                    "$COMMON/pb_encode_with_malloc.o",
+                    "$COMMON/pb_decode_with_malloc.o",
+                    "$COMMON/malloc_wrappers.o"])
+
+env.RunTest(fuzz)
+
+fuzzstub = malloc_env.Program(["fuzzstub.c",
+                    "alltypes_pointer.pb.c",
+                    "alltypes_static.pb.c",
+                    "$COMMON/pb_encode_with_malloc.o",
+                    "$COMMON/pb_decode_with_malloc.o",
+                    "$COMMON/malloc_wrappers.o"])
+
+
--- a/tests/fuzztest/alltypes_pointer.options
+++ b/tests/fuzztest/alltypes_pointer.options
@@ -0,0 +1,3 @@
+# Generate all fields as pointers.
+* type:FT_POINTER
+
--- a/tests/fuzztest/alltypes_static.options
+++ b/tests/fuzztest/alltypes_static.options
@@ -0,0 +1,3 @@
+* max_size:32
+* max_count:8
+*.extensions type:FT_IGNORE
--- a/tests/fuzztest/fuzzstub.c
+++ b/tests/fuzztest/fuzzstub.c
@@ -0,0 +1,189 @@
+/* Fuzz testing for the nanopb core.
+ * This can be used with external fuzzers, e.g. radamsa.
+ * It performs most of the same checks as fuzztest, but does not feature data generation.
+ */
+
+#include <pb_decode.h>
+#include <pb_encode.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <assert.h>
+#include <time.h>
+#include <malloc_wrappers.h>
+#include "alltypes_static.pb.h"
+#include "alltypes_pointer.pb.h"
+
+#define BUFSIZE 4096
+
+static bool do_static_decode(uint8_t *buffer, size_t msglen, bool assert_success)
+{
+    pb_istream_t stream;
+    bool status;
+    
+    alltypes_static_AllTypes *msg = malloc_with_check(sizeof(alltypes_static_AllTypes));
+    stream = pb_istream_from_buffer(buffer, msglen);
+    status = pb_decode(&stream, alltypes_static_AllTypes_fields, msg);
+    
+    if (!status && assert_success)
+    {
+        /* Anything that was successfully encoded, should be decodeable.
+         * One exception: strings without null terminator are encoded up
+         * to end of buffer, but refused on decode because the terminator
+         * would not fit. */
+        if (strcmp(stream.errmsg, "string overflow") != 0)
+            assert(status);
+    }
+    
+    free_with_check(msg);
+    return status;
+}
+
+static bool do_pointer_decode(uint8_t *buffer, size_t msglen, bool assert_success)
+{
+    pb_istream_t stream;
+    bool status;
+    alltypes_pointer_AllTypes *msg;
+    
+    msg = malloc_with_check(sizeof(alltypes_pointer_AllTypes));
+    memset(msg, 0, sizeof(alltypes_pointer_AllTypes));
+    stream = pb_istream_from_buffer(buffer, msglen);
+
+    assert(get_alloc_count() == 0);
+    status = pb_decode(&stream, alltypes_pointer_AllTypes_fields, msg);
+    
+    if (assert_success)
+        assert(status);
+    
+    pb_release(alltypes_pointer_AllTypes_fields, msg);    
+    assert(get_alloc_count() == 0);
+    
+    free_with_check(msg);
+
+    return status;
+}
+
+/* Do a decode -> encode -> decode -> encode roundtrip */
+static void do_static_roundtrip(uint8_t *buffer, size_t msglen)
+{
+    bool status;
+    uint8_t *buf2 = malloc_with_check(BUFSIZE);
+    uint8_t *buf3 = malloc_with_check(BUFSIZE);
+    size_t msglen2, msglen3;
+    alltypes_static_AllTypes *msg1 = malloc_with_check(sizeof(alltypes_static_AllTypes));
+    alltypes_static_AllTypes *msg2 = malloc_with_check(sizeof(alltypes_static_AllTypes));
+    memset(msg1, 0, sizeof(alltypes_static_AllTypes));
+    memset(msg2, 0, sizeof(alltypes_static_AllTypes));
+    
+    {
+        pb_istream_t stream = pb_istream_from_buffer(buffer, msglen);
+        status = pb_decode(&stream, alltypes_static_AllTypes_fields, msg1);
+        assert(status);
+    }
+    
+    {
+        pb_ostream_t stream = pb_ostream_from_buffer(buf2, BUFSIZE);
+        status = pb_encode(&stream, alltypes_static_AllTypes_fields, msg1);
+        assert(status);
+        msglen2 = stream.bytes_written;
+    }
+    
+    {
+        pb_istream_t stream = pb_istream_from_buffer(buf2, msglen2);
+        status = pb_decode(&stream, alltypes_static_AllTypes_fields, msg2);
+        assert(status);
+    }
+    
+    {
+        pb_ostream_t stream = pb_ostream_from_buffer(buf3, BUFSIZE);
+        status = pb_encode(&stream, alltypes_static_AllTypes_fields, msg2);
+        assert(status);
+        msglen3 = stream.bytes_written;
+    }
+    
+    assert(msglen2 == msglen3);
+    assert(memcmp(buf2, buf3, msglen2) == 0);
+    
+    free_with_check(msg1);
+    free_with_check(msg2);
+    free_with_check(buf2);
+    free_with_check(buf3);
+}
+
+/* Do decode -> encode -> decode -> encode roundtrip */
+static void do_pointer_roundtrip(uint8_t *buffer, size_t msglen)
+{
+    bool status;
+    uint8_t *buf2 = malloc_with_check(BUFSIZE);
+    uint8_t *buf3 = malloc_with_check(BUFSIZE);
+    size_t msglen2, msglen3;
+    alltypes_pointer_AllTypes *msg1 = malloc_with_check(sizeof(alltypes_pointer_AllTypes));
+    alltypes_pointer_AllTypes *msg2 = malloc_with_check(sizeof(alltypes_pointer_AllTypes));
+    memset(msg1, 0, sizeof(alltypes_pointer_AllTypes));
+    memset(msg2, 0, sizeof(alltypes_pointer_AllTypes));
+    
+    {
+        pb_istream_t stream = pb_istream_from_buffer(buffer, msglen);
+        status = pb_decode(&stream, alltypes_pointer_AllTypes_fields, msg1);
+        assert(status);
+    }
+    
+    {
+        pb_ostream_t stream = pb_ostream_from_buffer(buf2, BUFSIZE);
+        status = pb_encode(&stream, alltypes_pointer_AllTypes_fields, msg1);
+        assert(status);
+        msglen2 = stream.bytes_written;
+    }
+    
+    {
+        pb_istream_t stream = pb_istream_from_buffer(buf2, msglen2);
+        status = pb_decode(&stream, alltypes_pointer_AllTypes_fields, msg2);
+        assert(status);
+    }
+    
+    {
+        pb_ostream_t stream = pb_ostream_from_buffer(buf3, BUFSIZE);
+        status = pb_encode(&stream, alltypes_pointer_AllTypes_fields, msg2);
+        assert(status);
+        msglen3 = stream.bytes_written;
+    }
+    
+    assert(msglen2 == msglen3);
+    assert(memcmp(buf2, buf3, msglen2) == 0);
+    
+    pb_release(alltypes_pointer_AllTypes_fields, msg1);
+    pb_release(alltypes_pointer_AllTypes_fields, msg2);
+    free_with_check(msg1);
+    free_with_check(msg2);
+    free_with_check(buf2);
+    free_with_check(buf3);
+}
+
+static void run_iteration()
+{
+    uint8_t *buffer = malloc_with_check(BUFSIZE);
+    size_t msglen;
+    bool status;
+    
+    msglen = fread(buffer, BUFSIZE, 1, stdin);
+
+    status = do_static_decode(buffer, msglen, false);
+    
+    if (status)
+        do_static_roundtrip(buffer, msglen);
+    
+    status = do_pointer_decode(buffer, msglen, false);
+    
+    if (status)
+        do_pointer_roundtrip(buffer, msglen);
+    
+    free_with_check(buffer);
+}
+
+int main(int argc, char **argv)
+{
+    run_iteration();
+    
+    return 0;
+}
+
--- a/tests/fuzztest/fuzztest.c
+++ b/tests/fuzztest/fuzztest.c
@@ -0,0 +1,431 @@
+/* Fuzz testing for the nanopb core.
+ * Attempts to verify all the properties defined in the security model document.
+ */
+
+#include <pb_decode.h>
+#include <pb_encode.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <assert.h>
+#include <time.h>
+#include <malloc_wrappers.h>
+#include "alltypes_static.pb.h"
+#include "alltypes_pointer.pb.h"
+
+static uint64_t random_seed;
+
+/* Uses xorshift64 here instead of rand() for both speed and
+ * reproducibility across platforms. */
+static uint32_t rand_word()
+{
+    random_seed ^= random_seed >> 12;
+    random_seed ^= random_seed << 25;
+    random_seed ^= random_seed >> 27;
+    return random_seed * 2685821657736338717ULL;
+}
+
+/* Get a random integer in range, with approximately flat distribution. */
+static int rand_int(int min, int max)
+{
+    return rand_word() % (max + 1 - min) + min;
+}
+
+static bool rand_bool()
+{
+    return rand_word() & 1;
+}
+
+/* Get a random byte, with skewed distribution.
+ * Important corner cases like 0xFF, 0x00 and 0xFE occur more
+ * often than other values. */
+static uint8_t rand_byte()
+{
+    uint32_t w = rand_word();
+    uint8_t b = w & 0xFF;
+    if (w & 0x100000)
+        b >>= (w >> 8) & 7;
+    if (w & 0x200000)
+        b <<= (w >> 12) & 7;
+    if (w & 0x400000)
+        b ^= 0xFF;
+    return b;
+}
+
+/* Get a random length, with skewed distribution.
+ * Favors the shorter lengths, but always atleast 1. */
+static size_t rand_len(size_t max)
+{
+    uint32_t w = rand_word();
+    size_t s;
+    if (w & 0x800000)
+        w &= 3;
+    else if (w & 0x400000)
+        w &= 15;
+    else if (w & 0x200000)
+        w &= 255;
+
+    s = (w % max);
+    if (s == 0)
+        s = 1;
+    
+    return s;
+}
+
+/* Fills a buffer with random data with skewed distribution. */
+static void rand_fill(uint8_t *buf, size_t count)
+{
+    while (count--)
+        *buf++ = rand_byte();
+}
+
+/* Fill with random protobuf-like data */
+static size_t rand_fill_protobuf(uint8_t *buf, size_t min_bytes, size_t max_bytes, int min_tag)
+{
+    pb_ostream_t stream = pb_ostream_from_buffer(buf, max_bytes);
+
+    while(stream.bytes_written < min_bytes)
+    {
+        pb_wire_type_t wt = rand_int(0, 3);
+        if (wt == 3) wt = 5; /* Gap in values */
+        
+        if (!pb_encode_tag(&stream, wt, rand_int(min_tag, min_tag + 512)))
+            break;
+    
+        if (wt == PB_WT_VARINT)
+        {
+            uint64_t value;
+            rand_fill((uint8_t*)&value, sizeof(value));
+            pb_encode_varint(&stream, value);
+        }
+        else if (wt == PB_WT_64BIT)
+        {
+            uint64_t value;
+            rand_fill((uint8_t*)&value, sizeof(value));
+            pb_encode_fixed64(&stream, &value);
+        }
+        else if (wt == PB_WT_32BIT)
+        {
+            uint32_t value;
+            rand_fill((uint8_t*)&value, sizeof(value));
+            pb_encode_fixed32(&stream, &value);
+        }
+        else if (wt == PB_WT_STRING)
+        {
+            size_t len;
+            uint8_t *buf;
+            
+            if (min_bytes > stream.bytes_written)
+                len = rand_len(min_bytes - stream.bytes_written);
+            else
+                len = 0;
+            
+            buf = malloc(len);
+            pb_encode_varint(&stream, len);
+            rand_fill(buf, len);
+            pb_write(&stream, buf, len);
+            free(buf);
+        }
+    }
+    
+    return stream.bytes_written;
+}
+
+/* Given a buffer of data, mess it up a bit */
+static void rand_mess(uint8_t *buf, size_t count)
+{
+    int m = rand_int(0, 3);
+    
+    if (m == 0)
+    {
+        /* Replace random substring */
+        int s = rand_int(0, count - 1);
+        int l = rand_len(count - s);
+        rand_fill(buf + s, l);
+    }
+    else if (m == 1)
+    {
+        /* Swap random bytes */
+        int a = rand_int(0, count - 1);
+        int b = rand_int(0, count - 1);
+        int x = buf[a];
+        buf[a] = buf[b];
+        buf[b] = x;
+    }
+    else if (m == 2)
+    {
+        /* Duplicate substring */
+        int s = rand_int(0, count - 2);
+        int l = rand_len((count - s) / 2);
+        memcpy(buf + s + l, buf + s, l);
+    }
+    else if (m == 3)
+    {
+        /* Add random protobuf noise */
+        int s = rand_int(0, count - 1);
+        int l = rand_len(count - s);
+        rand_fill_protobuf(buf + s, l, count - s, 1);
+    }
+}
+
+/* Some default data to put in the message */
+static const alltypes_static_AllTypes initval = alltypes_static_AllTypes_init_default;
+
+#define BUFSIZE 4096
+
+static bool do_static_encode(uint8_t *buffer, size_t *msglen)
+{
+    pb_ostream_t stream;
+    bool status;
+
+    /* Allocate a message and fill it with defaults */
+    alltypes_static_AllTypes *msg = malloc_with_check(sizeof(alltypes_static_AllTypes));
+    memcpy(msg, &initval, sizeof(initval));
+
+    /* Apply randomness to the data before encoding */
+    while (rand_int(0, 7))
+        rand_mess((uint8_t*)msg, sizeof(alltypes_static_AllTypes));
+
+    stream = pb_ostream_from_buffer(buffer, BUFSIZE);
+    status = pb_encode(&stream, alltypes_static_AllTypes_fields, msg);
+    assert(stream.bytes_written <= BUFSIZE);
+    assert(stream.bytes_written <= alltypes_static_AllTypes_size);
+    
+    *msglen = stream.bytes_written;
+    free_with_check(msg);
+    
+    return status;
+}
+
+/* Append or prepend protobuf noise */
+static void do_protobuf_noise(uint8_t *buffer, size_t *msglen)
+{
+    int m = rand_int(0, 2);
+    size_t max_size = BUFSIZE - 32 - *msglen;
+    if (m == 1)
+    {
+        /* Prepend */
+        uint8_t *tmp = malloc_with_check(BUFSIZE);
+        size_t s = rand_fill_protobuf(tmp, rand_len(max_size), BUFSIZE - *msglen, 512);
+        memmove(buffer + s, buffer, *msglen);
+        memcpy(buffer, tmp, s);
+        free_with_check(tmp);
+        *msglen += s;
+    }
+    else if (m == 2)
+    {
+        /* Append */
+        size_t s = rand_fill_protobuf(buffer + *msglen, rand_len(max_size), BUFSIZE - *msglen, 512);
+        *msglen += s;
+    }
+}
+
+static bool do_static_decode(uint8_t *buffer, size_t msglen, bool assert_success)
+{
+    pb_istream_t stream;
+    bool status;
+    
+    alltypes_static_AllTypes *msg = malloc_with_check(sizeof(alltypes_static_AllTypes));
+    rand_fill((uint8_t*)msg, sizeof(alltypes_static_AllTypes));
+    stream = pb_istream_from_buffer(buffer, msglen);
+    status = pb_decode(&stream, alltypes_static_AllTypes_fields, msg);
+    
+    if (!status && assert_success)
+    {
+        /* Anything that was successfully encoded, should be decodeable.
+         * One exception: strings without null terminator are encoded up
+         * to end of buffer, but refused on decode because the terminator
+         * would not fit. */
+        if (strcmp(stream.errmsg, "string overflow") != 0)
+            assert(status);
+    }
+    
+    free_with_check(msg);
+    return status;
+}
+
+static bool do_pointer_decode(uint8_t *buffer, size_t msglen, bool assert_success)
+{
+    pb_istream_t stream;
+    bool status;
+    alltypes_pointer_AllTypes *msg;
+    
+    msg = malloc_with_check(sizeof(alltypes_pointer_AllTypes));
+    memset(msg, 0, sizeof(alltypes_pointer_AllTypes));
+    stream = pb_istream_from_buffer(buffer, msglen);
+
+    assert(get_alloc_count() == 0);
+    status = pb_decode(&stream, alltypes_pointer_AllTypes_fields, msg);
+    
+    if (assert_success)
+        assert(status);
+    
+    pb_release(alltypes_pointer_AllTypes_fields, msg);    
+    assert(get_alloc_count() == 0);
+    
+    free_with_check(msg);
+
+    return status;
+}
+
+/* Do a decode -> encode -> decode -> encode roundtrip */
+static void do_static_roundtrip(uint8_t *buffer, size_t msglen)
+{
+    bool status;
+    uint8_t *buf2 = malloc_with_check(BUFSIZE);
+    uint8_t *buf3 = malloc_with_check(BUFSIZE);
+    size_t msglen2, msglen3;
+    alltypes_static_AllTypes *msg1 = malloc_with_check(sizeof(alltypes_static_AllTypes));
+    alltypes_static_AllTypes *msg2 = malloc_with_check(sizeof(alltypes_static_AllTypes));
+    memset(msg1, 0, sizeof(alltypes_static_AllTypes));
+    memset(msg2, 0, sizeof(alltypes_static_AllTypes));
+    
+    {
+        pb_istream_t stream = pb_istream_from_buffer(buffer, msglen);
+        status = pb_decode(&stream, alltypes_static_AllTypes_fields, msg1);
+        assert(status);
+    }
+    
+    {
+        pb_ostream_t stream = pb_ostream_from_buffer(buf2, BUFSIZE);
+        status = pb_encode(&stream, alltypes_static_AllTypes_fields, msg1);
+        assert(status);
+        msglen2 = stream.bytes_written;
+    }
+    
+    {
+        pb_istream_t stream = pb_istream_from_buffer(buf2, msglen2);
+        status = pb_decode(&stream, alltypes_static_AllTypes_fields, msg2);
+        assert(status);
+    }
+    
+    {
+        pb_ostream_t stream = pb_ostream_from_buffer(buf3, BUFSIZE);
+        status = pb_encode(&stream, alltypes_static_AllTypes_fields, msg2);
+        assert(status);
+        msglen3 = stream.bytes_written;
+    }
+    
+    assert(msglen2 == msglen3);
+    assert(memcmp(buf2, buf3, msglen2) == 0);
+    
+    free_with_check(msg1);
+    free_with_check(msg2);
+    free_with_check(buf2);
+    free_with_check(buf3);
+}
+
+/* Do decode -> encode -> decode -> encode roundtrip */
+static void do_pointer_roundtrip(uint8_t *buffer, size_t msglen)
+{
+    bool status;
+    uint8_t *buf2 = malloc_with_check(BUFSIZE);
+    uint8_t *buf3 = malloc_with_check(BUFSIZE);
+    size_t msglen2, msglen3;
+    alltypes_pointer_AllTypes *msg1 = malloc_with_check(sizeof(alltypes_pointer_AllTypes));
+    alltypes_pointer_AllTypes *msg2 = malloc_with_check(sizeof(alltypes_pointer_AllTypes));
+    memset(msg1, 0, sizeof(alltypes_pointer_AllTypes));
+    memset(msg2, 0, sizeof(alltypes_pointer_AllTypes));
+    
+    {
+        pb_istream_t stream = pb_istream_from_buffer(buffer, msglen);
+        status = pb_decode(&stream, alltypes_pointer_AllTypes_fields, msg1);
+        assert(status);
+    }
+    
+    {
+        pb_ostream_t stream = pb_ostream_from_buffer(buf2, BUFSIZE);
+        status = pb_encode(&stream, alltypes_pointer_AllTypes_fields, msg1);
+        assert(status);
+        msglen2 = stream.bytes_written;
+    }
+    
+    {
+        pb_istream_t stream = pb_istream_from_buffer(buf2, msglen2);
+        status = pb_decode(&stream, alltypes_pointer_AllTypes_fields, msg2);
+        assert(status);
+    }
+    
+    {
+        pb_ostream_t stream = pb_ostream_from_buffer(buf3, BUFSIZE);
+        status = pb_encode(&stream, alltypes_pointer_AllTypes_fields, msg2);
+        assert(status);
+        msglen3 = stream.bytes_written;
+    }
+    
+    assert(msglen2 == msglen3);
+    assert(memcmp(buf2, buf3, msglen2) == 0);
+    
+    pb_release(alltypes_pointer_AllTypes_fields, msg1);
+    pb_release(alltypes_pointer_AllTypes_fields, msg2);
+    free_with_check(msg1);
+    free_with_check(msg2);
+    free_with_check(buf2);
+    free_with_check(buf3);
+}
+
+static void run_iteration()
+{
+    uint8_t *buffer = malloc_with_check(BUFSIZE);
+    size_t msglen;
+    bool status;
+    
+    rand_fill(buffer, BUFSIZE);
+
+    if (do_static_encode(buffer, &msglen))
+    {
+        do_protobuf_noise(buffer, &msglen);
+    
+        status = do_static_decode(buffer, msglen, true);
+        
+        if (status)
+            do_static_roundtrip(buffer, msglen);
+        
+        status = do_pointer_decode(buffer, msglen, true);
+        
+        if (status)
+            do_pointer_roundtrip(buffer, msglen);
+        
+        /* Apply randomness to the encoded data */
+        while (rand_bool())
+            rand_mess(buffer, BUFSIZE);
+        
+        /* Apply randomness to encoded data length */
+        if (rand_bool())
+            msglen = rand_int(0, BUFSIZE);
+        
+        status = do_static_decode(buffer, msglen, false);
+        do_pointer_decode(buffer, msglen, status);
+        
+        if (status)
+        {
+            do_static_roundtrip(buffer, msglen);
+            do_pointer_roundtrip(buffer, msglen);
+        }
+    }
+    
+    free_with_check(buffer);
+}
+
+int main(int argc, char **argv)
+{
+    int i;
+    if (argc > 1)
+    {
+        random_seed = atol(argv[1]);
+    }
+    else
+    {
+        random_seed = time(NULL);
+    }
+    
+    fprintf(stderr, "Random seed: %llu\n", (long long unsigned)random_seed);
+    
+    for (i = 0; i < 10000; i++)
+    {
+        run_iteration();
+    }
+    
+    return 0;
+}
+
--- a/tests/fuzztest/run_radamsa.sh
+++ b/tests/fuzztest/run_radamsa.sh
@@ -0,0 +1,12 @@
+#!/bin/bash
+
+TMP=`tempfile`
+
+echo $TMP
+while true
+do
+   radamsa sample_data/* > $TMP
+   $1 < $TMP
+   test $? -gt 127 && break
+done
+ 
--- a/tests/fuzztest/sample_data/sample1.pb
+++ b/tests/fuzztest/sample_data/sample1.pb
--- a/tests/fuzztest/sample_data/sample2.pb
+++ b/tests/fuzztest/sample_data/sample2.pb
--- a/tests/mem_release/SConscript
+++ b/tests/mem_release/SConscript
@@ -0,0 +1,12 @@
+Import("env", "malloc_env")
+
+env.NanopbProto("mem_release.proto")
+
+test = malloc_env.Program(["mem_release.c",
+                    "mem_release.pb.c",
+                    "$COMMON/pb_encode_with_malloc.o",
+                    "$COMMON/pb_decode_with_malloc.o",
+                    "$COMMON/malloc_wrappers.o"])
+
+env.RunTest(test)
+
--- a/tests/mem_release/mem_release.c
+++ b/tests/mem_release/mem_release.c
@@ -0,0 +1,106 @@
+/* Make sure that all fields are freed in various scenarios. */
+
+#include <pb_decode.h>
+#include <pb_encode.h>
+#include <malloc_wrappers.h>
+#include <stdio.h>
+#include <test_helpers.h>
+#include "mem_release.pb.h"
+
+#define TEST(x) if (!(x)) { \
+    fprintf(stderr, "Test " #x " on line %d failed.\n", __LINE__); \
+    return false; \
+    }
+
+static char *test_str_arr[] = {"1", "2", ""};
+static SubMessage test_msg_arr[] = {SubMessage_init_zero, SubMessage_init_zero};
+
+static bool do_test()
+{
+    uint8_t buffer[256];
+    size_t msgsize;
+    
+    /* Construct a message with various fields filled in */
+    {
+        TestMessage msg = TestMessage_init_zero;
+        pb_extension_t ext2;
+        pb_ostream_t stream;
+        msg.static_req_submsg.dynamic_str = "12345";
+        msg.static_req_submsg.dynamic_str_arr_count = 3;
+        msg.static_req_submsg.dynamic_str_arr = test_str_arr;
+        msg.static_req_submsg.dynamic_submsg_count = 2;
+        msg.static_req_submsg.dynamic_submsg = test_msg_arr;
+        msg.static_req_submsg.dynamic_submsg[1].dynamic_str = "abc";
+        msg.static_opt_submsg.dynamic_str = "abc";
+        msg.has_static_opt_submsg = true;
+        msg.dynamic_submsg = &msg.static_req_submsg;
+        
+        msg.extensions = &ext2;
+        ext2.type = &static_ext;
+        ext2.dest = &msg.static_req_submsg;
+        ext2.next = NULL;
+        
+        stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
+        if (!pb_encode(&stream, TestMessage_fields, &msg))
+        {
+            fprintf(stderr, "Encode failed: %s\n", PB_GET_ERROR(&stream));
+            return false;
+        }
+        msgsize = stream.bytes_written;
+    }
+    
+    /* Output encoded message for debug */
+    SET_BINARY_MODE(stdout);
+    fwrite(buffer, 1, msgsize, stdout);
+    
+    /* Decode memory using dynamic allocation */
+    {
+        TestMessage msg = TestMessage_init_zero;
+        pb_istream_t stream;
+        SubMessage ext2_dest = SubMessage_init_zero;
+        pb_extension_t ext2;
+        
+        msg.extensions = &ext2;
+        ext2.type = &static_ext;
+        ext2.dest = &ext2_dest;
+        ext2.next = NULL;
+        
+        stream = pb_istream_from_buffer(buffer, msgsize);
+        if (!pb_decode(&stream, TestMessage_fields, &msg))
+        {
+            fprintf(stderr, "Decode failed: %s\n", PB_GET_ERROR(&stream));
+            return false;
+        }
+        
+        /* Make sure it encodes back to same data */
+        {
+            uint8_t buffer2[256];
+            pb_ostream_t ostream = pb_ostream_from_buffer(buffer2, sizeof(buffer2));
+            TEST(pb_encode(&ostream, TestMessage_fields, &msg));
+            TEST(ostream.bytes_written == msgsize);
+            TEST(memcmp(buffer, buffer2, msgsize) == 0);
+        }
+        
+        /* Make sure that malloc counters work */
+        TEST(get_alloc_count() > 0);
+        
+        /* Make sure that pb_release releases everything */
+        pb_release(TestMessage_fields, &msg);
+        TEST(get_alloc_count() == 0);
+        
+        /* Check that double-free is a no-op */
+        pb_release(TestMessage_fields, &msg);
+        TEST(get_alloc_count() == 0);
+    }
+    
+    return true;
+}
+
+int main()
+{
+    if (do_test())
+        return 0;
+    else
+        return 1;
+}
+
--- a/tests/mem_release/mem_release.proto
+++ b/tests/mem_release/mem_release.proto
@@ -0,0 +1,23 @@
+syntax = "proto2";
+import "nanopb.proto";
+
+message SubMessage
+{
+    optional string dynamic_str = 1 [(nanopb).type = FT_POINTER];
+    repeated string dynamic_str_arr = 2 [(nanopb).type = FT_POINTER];
+    repeated SubMessage dynamic_submsg = 3 [(nanopb).type = FT_POINTER];
+}
+
+message TestMessage
+{
+    required SubMessage static_req_submsg = 1 [(nanopb).type = FT_STATIC];
+    optional SubMessage dynamic_submsg = 2 [(nanopb).type = FT_POINTER];
+    optional SubMessage static_opt_submsg = 3 [(nanopb).type = FT_STATIC];
+    extensions 100 to 200;
+}
+
+extend TestMessage
+{
+    optional SubMessage static_ext = 101 [(nanopb).type = FT_STATIC];
+}
+
--- a/tests/missing_fields/missing_fields.c
+++ b/tests/missing_fields/missing_fields.c
@@ -8,6 +8,7 @@
 int main()
 {
    uint8_t buffer[512];
+    size_t size;
    
    /* Create a message with one missing field */
    {
@@ -19,12 +20,14 @@ int main()
            printf("Encode failed.\n");
            return 1;
        }
+
+        size = stream.bytes_written;
    }

    /* Test that it decodes properly if we don't require that field */
    {
        MissingField msg = {0};
-        pb_istream_t stream = pb_istream_from_buffer(buffer, sizeof(buffer));
+        pb_istream_t stream = pb_istream_from_buffer(buffer, size);
        
        if (!pb_decode(&stream, MissingField_fields, &msg))
        {
@@ -36,7 +39,7 @@ int main()
    /* Test that it does *not* decode properly if we require the field */
    {
        AllFields msg = {0};
-        pb_istream_t stream = pb_istream_from_buffer(buffer, sizeof(buffer));
+        pb_istream_t stream = pb_istream_from_buffer(buffer, size);
        
        if (pb_decode(&stream, AllFields_fields, &msg))
        {
--- a/tests/no_errmsg/SConscript
+++ b/tests/no_errmsg/SConscript
@@ -4,8 +4,6 @@ Import("env")

 # Take copy of the files for custom build.
 c = Copy("$TARGET", "$SOURCE")
-env.Command("pb_encode.c", "#../pb_encode.c", c)
-env.Command("pb_decode.c", "#../pb_decode.c", c)
 env.Command("alltypes.pb.h", "$BUILD/alltypes/alltypes.pb.h", c)
 env.Command("alltypes.pb.c", "$BUILD/alltypes/alltypes.pb.c", c)
 env.Command("encode_alltypes.c", "$BUILD/alltypes/encode_alltypes.c", c)
@@ -15,9 +13,15 @@ env.Command("decode_alltypes.c", "$BUILD/alltypes/decode_alltypes.c", c)
 opts = env.Clone()
 opts.Append(CPPDEFINES = {'PB_NO_ERRMSG': 1})

+# Build new version of core
+strict = opts.Clone()
+strict.Append(CFLAGS = strict['CORECFLAGS'])
+strict.Object("pb_decode_noerr.o", "$NANOPB/pb_decode.c")
+strict.Object("pb_encode_noerr.o", "$NANOPB/pb_encode.c")
+
 # Now build and run the test normally.
-enc = opts.Program(["encode_alltypes.c", "alltypes.pb.c", "pb_encode.c"])
-dec = opts.Program(["decode_alltypes.c", "alltypes.pb.c", "pb_decode.c"])
+enc = opts.Program(["encode_alltypes.c", "alltypes.pb.c", "pb_encode_noerr.o"])
+dec = opts.Program(["decode_alltypes.c", "alltypes.pb.c", "pb_decode_noerr.o"])

 env.RunTest(enc)
 env.RunTest([dec, "encode_alltypes.output"])
--- a/tests/options/options.expected
+++ b/tests/options/options.expected
@@ -5,3 +5,6 @@ pb_callback_t int32_callback;
 \sEnumValue1 = 1
 Message5_EnumValue1
 } pb_packed my_packed_struct;
+! skipped_field
+! SkippedMessage
+
--- a/tests/options/options.proto
+++ b/tests/options/options.proto
@@ -63,11 +63,15 @@ message my_packed_struct
 }

 // Message with ignored field
-// Note: doesn't really test if the field is missing in the output,
-// but atleast tests that the output compiles.
 message Message6
 {
    required int32 field1 = 1;
-    optional int32 field2 = 2 [(nanopb).type = FT_IGNORE];
+    optional int32 skipped_field = 2 [(nanopb).type = FT_IGNORE];
 }

+// Message that is skipped
+message SkippedMessage
+{
+    option (nanopb_msgopt).skip_message = true;
+    required int32 foo = 1;
+}
--- a/tests/package_name/SConscript
+++ b/tests/package_name/SConscript
@@ -0,0 +1,36 @@
+# Check that alltypes test case works also when the .proto file defines
+# a package name.
+
+Import("env")
+
+# Build a modified alltypes.proto
+def modify_proto(target, source, env):
+    '''Add a "package test.package;" directive to the beginning of the .proto file.'''
+    data = open(str(source[0]), 'r').read()
+    open(str(target[0]), 'w').write("package test.package;\n\n" + data)
+    return 0
+
+env.Command("alltypes.proto", "#alltypes/alltypes.proto", modify_proto)
+env.Command("alltypes.options", "#alltypes/alltypes.options", Copy("$TARGET", "$SOURCE"))
+env.NanopbProto(["alltypes", "alltypes.options"])
+
+# Build a modified encode_alltypes.c
+def modify_c(target, source, env):
+    '''Add package name to type names in .c file.'''
+
+    data = open(str(source[0]), 'r').read()
+    
+    type_names = ['AllTypes', 'MyEnum', 'HugeEnum']
+    for name in type_names:
+        data = data.replace(name, 'test_package_' + name)
+    
+    open(str(target[0]), 'w').write(data)
+    return 0
+env.Command("encode_alltypes.c", "#alltypes/encode_alltypes.c", modify_c)
+
+# Encode and compare results to original alltypes testcase
+enc = env.Program(["encode_alltypes.c", "alltypes.pb.c", "$COMMON/pb_encode.o"])
+refdec = "$BUILD/alltypes/decode_alltypes$PROGSUFFIX"
+env.RunTest(enc)
+env.Compare(["encode_alltypes.output", "$BUILD/alltypes/encode_alltypes.output"])
+
--- a/tests/regression/issue_125/SConscript
+++ b/tests/regression/issue_125/SConscript
@@ -0,0 +1,9 @@
+# Regression test for Issue 125: Wrong identifier name for extension fields
+
+Import("env")
+
+env.NanopbProto(["extensionbug", "extensionbug.options"])
+env.Object('extensionbug.pb.c')
+
+env.Match(['extensionbug.pb.h', 'extensionbug.expected'])
+
--- a/tests/regression/issue_125/extensionbug.expected
+++ b/tests/regression/issue_125/extensionbug.expected
@@ -0,0 +1,3 @@
+pb_extension_type_t Message2_extras
+uint32_t field2
+
--- a/tests/regression/issue_125/extensionbug.options
+++ b/tests/regression/issue_125/extensionbug.options
@@ -0,0 +1,4 @@
+* type:FT_IGNORE
+
+Message2.extras type:FT_STATIC
+Message2.field2 type:FT_STATIC
--- a/tests/regression/issue_125/extensionbug.proto
+++ b/tests/regression/issue_125/extensionbug.proto
@@ -0,0 +1,16 @@
+message Message1
+{
+	optional uint32	fieldA = 1;
+	extensions 30 to max;
+}
+
+message Message2
+{
+	extend Message1
+	{
+		optional Message2 extras = 30;
+	}
+	
+	optional uint32	field1 = 1;
+	optional uint32 field2 = 2;
+}
--- a/tests/site_scons/site_init.py
+++ b/tests/site_scons/site_init.py
@@ -19,19 +19,24 @@ def add_nanopb_builders(env):
        else:
            infile = None
        
+        if env.has_key("COMMAND"):
+            args = [env["COMMAND"]]
+        else:
            args = [str(source[0])]
+        
        if env.has_key('ARGS'):
            args.extend(env['ARGS'])
        
+        print 'Command line: ' + str(args)
        pipe = subprocess.Popen(args,
                                stdin = infile,
                                stdout = open(str(target[0]), 'w'),
                                stderr = sys.stderr)
        result = pipe.wait()
        if result == 0:
-            print '\033[32m[ OK ]\033[0m   Ran ' + str(source[0])
+            print '\033[32m[ OK ]\033[0m   Ran ' + args[0]
        else:
-            print '\033[31m[FAIL]\033[0m   Program ' + str(source[0]) + ' returned ' + str(result)
+            print '\033[31m[FAIL]\033[0m   Program ' + args[0] + ' returned ' + str(result)
        return result
        
    run_test_builder = Builder(action = run_test,
@@ -80,9 +85,20 @@ def add_nanopb_builders(env):
        data = open(str(source[0]), 'rU').read()
        patterns = open(str(source[1]))
        for pattern in patterns:
-            if pattern.strip() and not re.search(pattern.strip(), data, re.MULTILINE):
+            if pattern.strip():
+                invert = False
+                if pattern.startswith('! '):
+                    invert = True
+                    pattern = pattern[2:]
+                
+                status = re.search(pattern.strip(), data, re.MULTILINE)
+                
+                if not status and not invert:
                    print '\033[31m[FAIL]\033[0m   Pattern not found in ' + str(source[0]) + ': ' + pattern
                    return 1
+                elif status and invert:
+                    print '\033[31m[FAIL]\033[0m   Pattern should not exist, but does in ' + str(source[0]) + ': ' + pattern
+                    return 1
        else:
            print '\033[32m[ OK ]\033[0m   All patterns found in ' + str(source[0])
            return 0
--- a/tests/splint/SConscript
+++ b/tests/splint/SConscript
@@ -0,0 +1,13 @@
+# Check the nanopb core using splint
+
+Import('env')
+
+p = env.WhereIs('splint')
+
+if p:
+    env.Command('pb_decode.splint', '$NANOPB/pb_decode.c',
+        'splint -f splint/splint.rc $SOURCE 2> $TARGET')
+
+    env.Command('pb_encode.splint', '$NANOPB/pb_encode.c',
+        'splint -f splint/splint.rc $SOURCE 2> $TARGET')
+
--- a/tests/splint/splint.rc
+++ b/tests/splint/splint.rc
@@ -0,0 +1,37 @@
+checks
+partial
+matchanyintegral
+strictlib
+-isoreserved		# to be fixed in 0.3
+-nullassign
+-predboolint
+-predboolptr
+ptrnegate
+-switchloopbreak
+ignoresigns
+-infloopsuncon
+-type
+
+# splint's memory checks don't quite work without annotations
+-mustfreeonly
+-compmempass
+-nullret
+-observertrans
+-statictrans
+-compdestroy
+-nullpass
+-nullstate
+-compdef
+-usereleased
+-temptrans
+-dependenttrans
+-kepttrans
+-branchstate
+-immediatetrans
+
+# These tests give false positives, compiler typically has
+# better warnings for these.
+-noret
+-noeffect
+-usedef
+
Author	SHA1	Message	Date
Petteri Aimonen	eb66254b79	Publishing nanopb-0.2.9.2	2015-01-24 17:18:25 +02:00
Petteri Aimonen	641c743b27	Fix build failure due to missing dependency in SConscript	2014-12-27 00:38:22 +02:00
Petteri Aimonen	4ce729df7f	Backport memory leak fix for issue 138.	2014-12-27 00:01:45 +02:00
Petteri Aimonen	83157e3362	Backport mem_release testcase from 0.3.1. Also backports cdbf51 generator bugfix.	2014-12-26 23:59:56 +02:00
Petteri Aimonen	45fc9f3ef6	Publishing nanopb-0.2.9.1	2014-09-11 19:14:45 +03:00
Petteri Aimonen	115de6e555	Update changelog	2014-09-11 19:13:59 +03:00
Petteri Aimonen	9c92410e2a	Add a better fuzz test. Attempts to verify all the properties defined in the security model, while also being portable and able to run on many platforms.	2014-09-11 18:25:23 +03:00
Petteri Aimonen	6df566859d	Protect against size_t overflows in pb_dec_bytes/pb_dec_string. Possible consequences of bug: 1) Denial of service by causing a crash Possible when all of the following apply: - Untrusted data is passed to pb_decode() - The top-level message contains a static string field as the first field. Causes a single write of '0' byte to 1 byte before the message struct. 2) Remote code execution Possible when all of the following apply: - 64-bit platform - The message or a submessage contains a static/pointer string field. - Decoding directly from a custom pb_istream_t - bytes_left on the stream is set to larger than 4 GB Causes a write of up to 4 GB of data past the string field. 3) Possible heap corruption or remote code execution Possible when all of the following apply: - less than 64-bit platform - The message or a submessage contains a pointer-type bytes field. Causes a write of sizeof(pb_size_t) bytes of data past a 0-byte long malloc()ed buffer. On many malloc() implementations, this causes at most a crash. However, remote code execution through a controlled jump cannot be ruled out. -- Detailed analysis follows In the following consideration, I define "platform bitness" as equal to number of bits in size_t datatype. Therefore most 8-bit platforms are regarded as 16-bit for the purposes of this discussion. 1. The overflow in pb_dec_string The overflow happens in this computation: uint32_t size; size_t alloc_size; alloc_size = size + 1; There are two ways in which the overflow can occur: In the uint32_t addition, or in the cast to size_t. This depends on the platform bitness. On 32- and 64-bit platforms, the size has to be UINT32_MAX for the overflow to occur. In that case alloc_size will be 0. On 16-bit platforms, overflow will happen whenever size is more than UINT16_MAX, and resulting alloc_size is attacker controlled. For static fields, the alloc_size value is just checked against the field data size. For pointer fields, the alloc_size value is passed to malloc(). End result in both cases is the same, the storage is 0 or just a few bytes in length. On 16-bit platforms, another overflow occurs in the call to pb_read(), when passing the original size. An attacker will want the passed value to be larger than the alloc_size, therefore the only reasonable choice is to have size = UINT16_MAX and alloc_size = 0. Any larger multiple will truncate to the same values. At this point we have read atleast the tag and the string length of the message, i.e. atleast 3 bytes. The maximum initial value for stream bytes_left is SIZE_MAX, thus at this point at most SIZE_MAX-3 bytes are remaining. On 32-bit and 16-bit platforms this means that the size passed to pb_read() is always larger than the number of remaining bytes. This causes pb_read() to fail immediately, before reading any bytes. On 64-bit platforms, it is possible for the bytes_left value to be set to a value larger than UINT32_MAX, which is the wraparound point in size calculation. In this case pb_read() will succeed and write up to 4 GB of attacker controlled data over the RAM that comes after the string field. On all platforms, there is an unconditional write of a terminating null byte. Because the size of size_t typically reflects the size of the processor address space, a write at UINT16_MAX or UINT32_MAX bytes after the string field actually wraps back to before the string field. Consequently, on 32-bit and 16-bit platforms, the bug causes a single write of '0' byte at one byte before the string field. If the string field is in the middle of a message, this will just corrupt other data in the message struct. Because the message contents is attacker controlled anyway, this is a non-issue. However, if the string field is the first field in the top-level message, it can corrupt other data on the stack/heap before it. Typically a single '0' write at a location not controlled by attacker is enough only for a denial-of-service attack. When using pointer fields and malloc(), the attacker controlled alloc_size will cause a 0-size allocation to happen. By the same logic as before, on 32-bit and 16-bit platforms this causes a '0' byte write only. On 64-bit platforms, however, it will again allow up to 4 GB of malicious data to be written over memory, if the stream length allows the read. 2. The overflow in pb_dec_bytes This overflow happens in the PB_BYTES_ARRAY_T_ALLOCSIZE macro: The computation is done in size_t data type this time. This means that an overflow is possible only when n is larger than SIZE_MAX - offsetof(..). The offsetof value in this case is equal to sizeof(pb_size_t) bytes. Because the incoming size value is limited to 32 bits, no overflow can happen here on 64-bit platforms. The size will be passed to pb_read(). Like before, on 32-bit and 16-bit platforms the read will always fail before writing anything. This leaves only the write of bdest->size as exploitable. On statically allocated fields, the size field will always be allocated, regardless of alloc_size. In this case, no buffer overflow is possible here, but user code could possibly use the attacker controlled size value and read past a buffer. If the field is allocated through malloc(), this will allow a write of sizeof(pb_size_t) attacker controlled bytes to past a 0-byte long buffer. In typical malloc implementations, this will either fit in unused alignment padding area, or cause a heap corruption and a crash. Under very exceptional situation it could allow attacker to influence the behaviour of malloc(), possibly jumping into an attacker-controlled location and thus leading to remote code execution.	2014-09-11 18:25:23 +03:00
Petteri Aimonen	cc1c3a7963	Add just-to-be-sure check to allocate_field(). This check will help to detect bugs earlier, and is quite lightweight compared to malloc() anyway.	2014-09-11 18:25:23 +03:00
Petteri Aimonen	33585924de	Fix memory leak with duplicated fields and PB_ENABLE_MALLOC. If a required or optional field appeared twice in the message data, pb_decode will overwrite the old data with new one. That is fine, but with submessage fields, it didn't release the allocated subfields before overwriting. This bug can manifest if all of the following conditions are true: 1. There is a message with a "optional" or "required" submessage field that has type:FT_POINTER. 2. The submessage contains atleast one field with type:FT_POINTER. 3. The message data to be decoded has the submessage field twice in it.	2014-09-11 18:25:18 +03:00
Petteri Aimonen	b7add1e577	Fix crash in pb_release() if called twice on same message. There was a double-free bug in pb_release() because it didn't set size fields to zero after deallocation. Most commonly this happens if pb_decode() fails, internally calls pb_release() and then application code also calls pb_release().	2014-09-11 18:16:01 +03:00
Petteri Aimonen	2f05a35b5f	Publishing nanopb-0.2.9	2014-08-09 22:01:04 +03:00
Petteri Aimonen	4f76e64929	Update changelog	2014-08-04 19:13:39 +03:00
Petteri Aimonen	ec3bff4ba1	Generate #defines for initializing message structures. Usage like: MyMessage foo = MyMessage_init_default; MyMessage_init_default will initialize to default values defined in .proto. MyMessage_init_zero will initialize to null/zero values. Same results as {} or {0}, but will avoid compiler warnings by initializing everything explicitly. Update issue 79 Status: FixedInGit	2014-08-04 18:40:40 +03:00
Petteri Aimonen	1d7f60fec3	Add skip_message option to generator. Update issue 121 Status: FixedInGit	2014-07-20 14:56:12 +03:00
Petteri Aimonen	5749606f5d	Add support for inverted patterns in test framework.	2014-07-20 14:55:47 +03:00
Petteri Aimonen	eaa3c7b157	Cleanup and comment the code of network_server example. Update issue 123 Status: FixedInGit	2014-07-20 14:44:41 +03:00
Petteri Aimonen	3cf9668c75	Do not automatically add a dot with generator -e option. Now -e option in generator is more versatile. Especially it avoids double-dot problem with some build systems. Given foobar.proto, we now get: -e .pb => foobar.pb.c (default) -e _pb => foobar_pb.c -e '' => foobar.c Note that if you have used -e option previously, you will have to prepend . to the argument to get the same filenames as before. Update issue 122 Status: FixedInGit	2014-07-20 14:25:11 +03:00
Petteri Aimonen	7f97ad549e	Give better messages about the .options file path. Update issue 124 Status: FixedInGit	2014-07-20 14:18:21 +03:00
Petteri Aimonen	f2f9f8a9ed	Fix problem with .options file and extension fields. The options for an extension field were being looked up under wrong name (MessageName instead of MessageName.fieldname). Fixed the problem and added regression test. Created a new subfolder for regression test cases. Update issue 125 Status: FixedInGit	2014-07-20 14:02:56 +03:00
Petteri Aimonen	788d2825b0	Add unit tests for allocate_field().	2014-06-02 21:20:57 +03:00
Petteri Aimonen	99bc1d4f97	Make clearer that size = 0 in allocate_field() is not allowed. Back in design phase the code used realloc() for freeing the memory also. However, this is not entirely portable, and therefore the finished implementation used free() separately. There were some remnants of the size = 0 code in the allocate_field() code, which made it somewhat confusing. This change makes it clearer that size = 0 is not allowed (and not used by nanopb).	2014-06-02 21:12:38 +03:00
Petteri Aimonen	8a857a7f75	Don't use SIZE_MAX macro, as it is not in C89. Update issue 120 Status: FixedInGit	2014-06-02 21:09:06 +03:00
Petteri Aimonen	8611958a7f	Add PB_PACKED_STRUCT support for Keil MDK-ARM toolchain Patch from Jon Read. Update issue 119 Status: FixedInGit	2014-05-30 13:45:48 +03:00
Petteri Aimonen	2e9797af58	Setting version to 0.2.9-dev	2014-05-20 19:52:09 +03:00
Petteri Aimonen	2c51fb7771	Update changelog for 0.2.8	2014-05-20 19:46:48 +03:00
Petteri Aimonen	916bcb3643	Publishing nanopb-0.2.8	2014-05-20 19:35:00 +03:00
Petteri Aimonen	9cf788de54	Fix bug in alltypes test case that made fuzzing difficult.	2014-05-17 20:28:33 +03:00
Petteri Aimonen	5ef128616b	Fix security issue with PB_ENABLE_MALLOC. The multiplication in allocate_field could potentially overflow, leading to allocating too little memory. This could subsequently allow an attacker to cause a write past the buffer, overwriting other memory contents. The attack is possible if untrusted message data is decoded using nanopb, and the message type includes a pointer-type string or bytes field, or a repeated numeric field. Submessage fields are not affected. This issue only affects systems that have been compiled with PB_ENABLE_MALLOC enabled. Only version nanopb-0.2.7 is affected, as prior versions do not include this functionality. Update issue 117 Status: FixedInGit	2014-05-17 20:06:55 +03:00
Petteri Aimonen	ba2ab9ea65	Docs update, remove malloc from limitations list	2014-04-26 20:11:54 +03:00
Petteri Aimonen	e6a57e512f	Add option to not add timestamps to .pb.h and .pb.c preambles. Patch by rusnakp. Update issue 115 Status: FixedInGit	2014-04-18 15:40:40 +03:00
Petteri Aimonen	d177af1639	Fix typos in scons command line options	2014-04-15 20:30:50 +03:00
Petteri Aimonen	3b36235cef	Remove -O0 from tests CFLAGS so that optimized builds can be tested also	2014-04-15 20:27:38 +03:00
Petteri Aimonen	1d249a48ea	Fix bug in missing_fields test case	2014-04-09 19:39:12 +03:00
Petteri Aimonen	3e83d81b09	Use -fsanitize=undefined when running tests with clang	2014-04-09 19:28:57 +03:00
Petteri Aimonen	938c7ac3f3	Setting version to 0.2.8-dev	2014-04-07 20:45:04 +03:00
Petteri Aimonen	6d74c66ada	Publishing nanopb-0.2.7	2014-04-07 20:30:42 +03:00
Petteri Aimonen	c998ffe117	Update changelog	2014-04-07 20:30:12 +03:00
Petteri Aimonen	a8de6acf2d	Add rule for building coverage summary using lcov. Also modified a few tests to be more compatible with coverage information, so that they use the same pb_encode/decode.c instead of making a copy.	2014-04-05 13:26:37 +03:00
Petteri Aimonen	7880f308ea	Fix unused parameter warning when building without errmsg.	2014-04-05 13:25:44 +03:00
Petteri Aimonen	b63e582bdb	Add a convenience function pb_get_encoded_size() There is minimal size penalty from this, and it is probably much more intuitive to use than PB_OSTREAM_SIZING itself. This has been suggested before also, but I ended up refusing it back them. Reconsidering it now, I see that an intuitive API is much better than any amount of documentation explaining a non-intuitive API. Update issue 16 Status: FixedInGit	2014-04-05 11:26:39 +03:00
Petteri Aimonen	e5b855fec5	Add a 'found' field to pb_extension_t. Update issue 112 Status: FixedInGit	2014-04-05 11:11:05 +03:00
Petteri Aimonen	70dee34da6	Add some missing 'static' specifiers Update issue 91 Status: FixedInGit	2014-04-02 21:08:15 +03:00
Petteri Aimonen	99434724d0	Fix splint warnings, add splint test case	2014-04-02 21:07:30 +03:00
Petteri Aimonen	6c90e824c4	Fix compile error when default value given for extension field. Update issue 111 Status: FixedInGit	2014-04-01 16:47:53 +03:00
Petteri Aimonen	f4949119ad	Add stdlib.h to pb_syshdr.h for dynamic allocation	2014-03-18 16:13:54 +02:00
Petteri Aimonen	607cb998b5	More configuration options for dynamic alloc	2014-03-17 17:25:58 +02:00
Petteri Aimonen	ab62402059	Documentation updates	2014-03-16 15:52:19 +02:00
Petteri Aimonen	108864963f	Finish the alltypes_pointer testcase, use valgrind if available.	2014-03-15 09:39:27 +02:00
Petteri Aimonen	9be2cfe968	Get rid of pb_bytes_ptr_t, just allocate pb_bytes_array_t dynamically. This makes the internal logic much simpler, and also keeps the datatypes more similar between STATIC/POINTER cases. It will still be a bit cumbersome to use because of variable length array member. Macros PB_BYTES_ARRAY_T(n) and PB_BYTES_ARRAY_T_ALLOCSIZE(n) have been added to make life a bit easier. This has the drawback that it is no longer as easy to use externally allocated byte array as input for bytes field in pointer mode. However, this is still easy to do using callbacks, so it shouldn't be a large issue.	2014-03-15 08:45:58 +02:00
Petteri Aimonen	9c196b89ba	Add pb_release() function	2014-03-12 21:08:35 +02:00
Petteri Aimonen	bf61d2337b	More fixes for dynamic allocation	2014-03-10 18:19:38 +02:00
Petteri Aimonen	48ac461372	Bugfixes for dynamic allocation	2014-02-25 19:58:11 +02:00
Petteri Aimonen	011a30af9c	Beginnings of malloc support in pb_decode	2014-02-24 21:09:25 +02:00
Petteri Aimonen	842c960d5d	Setting version to 0.2.7-dev	2014-02-15 17:15:54 +02:00
Petteri Aimonen	7f397b067f	Publishing nanopb-0.2.6	2014-02-15 16:40:38 +02:00
Petteri Aimonen	586777b52f	Update changelog	2014-02-12 19:50:52 +02:00
Petteri Aimonen	542463dbaa	Move descriptor.proto to google/protobuf directory where it belongs. This fixes some interoperability problems when trying to compile nanopb.proto for e.g. the Google's C++ library. Update issue 104 Status: FixedInGit	2014-02-12 19:43:04 +02:00
Petteri Aimonen	fe0bf121eb	Add test case for package names defined in .proto file.	2014-02-12 19:22:01 +02:00
Petteri Aimonen	bd22cf2777	Improve status/error reporting in generator. Update issue 105 Status: FixedInGit	2014-02-12 18:51:48 +02:00
Petteri Aimonen	057165966c	Add AllTypes decoding test with callbacks. Update issue 99 Status: FixedInGit	2014-02-10 21:00:26 +02:00
Petteri Aimonen	235219a295	Add AllTypes encoding test for callbacks (issue #99 ) Still need to do the decoding side.	2014-02-09 19:28:50 +02:00
Petteri Aimonen	d3ed0744d0	Extreme negative integer constants are a bit nasty to define in C, just avoid the trouble in tests for now.	2014-02-05 17:56:37 +02:00
Petteri Aimonen	879860be19	Add coments to places where STATIC_ASSERT is used. Update issue 96 Status: FixedInGit	2014-02-04 20:34:57 +02:00
Petteri Aimonen	a2f8112166	Add proper suffixes for integer default values. Update issue 102 Status: FixedInGit	2014-02-04 20:24:44 +02:00
Petteri Aimonen	5efeb392e0	Add strict-aliasing GCC warnings for core	2014-02-04 20:11:52 +02:00
Petteri Aimonen	a46ed9f475	Fix problem with callback-type fields (issue #99 ).	2014-01-03 11:17:39 +02:00
Petteri Aimonen	843fc4b11a	Setting version to 0.2.6-dev	2014-01-01 11:44:47 +02:00