Stats about bigcode dataset:
Permissive licenses only :
Language
Raw (only exact dedup)
Near dedup
Near dedup + content filters
Near dedup + more near-dedup (1)
Near dedup + comments filter (1)
Near dedup + more near-dedup + comments(1)
Python
200 GB
80 GB
75.61 GB
61.97 GB
65 GB
(?) less than 60
Java
266 GB
112 GB110 GB
88 GB
92 GB
(?) less than 80
JavaScript
496 GB
166 GB
83 GB
65 GB
75 GB
(?) less than 60
C
255 GB
75 GB
73 GB (2)
C++
215 GB
65 GB
61 GB (2)