It is embarrassing for glibc not to check for overflow in calloc implementation prior to 2002. It is not only a security flaw but also violation of C Standards (even the first version ratified in 1989, usually referred to as C89).
The standard reads as follows:
void *calloc(size_t nmemb, size_t size);
The calloc function allocates space for an array of nmemb objects, each of whose size is size.[...]
and,
The calloc function returns either a null pointer or a pointer to the allocated space.
So if it cannot allocate space for an array of nmemb objects, each of whose size is size, then it has to return null pointer.
The code in question violates the newer C standards (C99, C11). For instance, it doesn't even compile using "gcc -std=c99 -pedantic-errors".
In the older C standard (C89), the code has undefined behaviour because implicit declaration of the function malloc is not compatible with its definition in the standard library. It is a futile task to try to define the undefined.
The standard reads as follows:
and,
So if it cannot allocate space for an array of nmemb objects, each of whose size is size, then it has to return null pointer.