Ancient sources discuss how much the rope should be pretensioned on the stretcher. The first mention is from Philon (Bel. 54-55):
"Make the diameter of the thickness of the cord 1/4D minus 1/12D (=1/6D); but stretch it, when the engine is being strung, until one third of the thickness is removed." (Marsden 1971: 115)
Second is from Heron (Bel. 108):
"Taking the other end through the opposite hole to the nearby windlass, we shall stretch it until the thickness of the fibre of the spring-cord has been reduced by a third" (Marsden 1971: 37)
According to Karpowicz (2008: 59) and Baker (2000d: 108) sinew can stretch safely only to 1,05 - 1,1 times it's original length (i.e. 5-10%). Nylon's characteristics seem to be in the same category with 115% stretch before breakage (e.g. McKenna et al 2004: 55). Based on these numbers it will not be possible to reduce the diameter of the fibers by 1/3. Let's assume the length (i.e. height) of a piece of fiber is h and it's radius is r. For the sake of simplicity, we also assume it's a perfectly uniform cylinder with volume V:
V = π*r²*h
As the fiber is being stretched, it's volume stays the same, but it's length increases. Thus the fiber stretched to 2/3 diameter can be described as
V = π*((2/3)r)²*xh
where x is a yet unknown modifier to the fiber's length required to keep the volume V the same. Now we solve x:
π*r²*h = π*((2/3)r)²*xh ||:π r²*h = ((2/3)r)²*xh ||:h r² = ((2/3)r)²*x r² = 4r²/9*x ||*9 9r² = 4r² *x ||:4r² x = (9r^2)/(4r^2) x = 9/4
So, in order to reduce the diameter of a cylinder to 2/3, it's length would have to be 9/4 of the original. In other words, the cylinder would have to stretch to 2,25 times of it's original length, which is way too much for both nylon and sinew.
However, it seems that reducing the diameter of a sinew rope to 2/3 is indeed possible, with polyester rope being very similar in stretching properties; nylon rope, however, refuses to stretch to 2/3 (Clift et al 2004: 59). I've tested the behavior of nylon in my own fairly crude tests. In one test I attached one end of a ~2.1mm nylon cord to a post and the other end to spring scale attached to a winch. The cord was then winched to 82.5kg (182 pounds) draw. Even at this point, there was no noticeable reduction in diameter as measured with calipers (1/10mm precision), yet the cord had stretched 12,3%. Estimating from manufacturer's break load numbers the cord should break around 80-106kg. Breaking loads in the same category (~120kg) were given for nylon in others sources, too. The cord used in this test was taken from the 4.85mm three-ply nylon cord used in the next test.
In another test, a three-ply twisted nylon cord was measured at zero load. Four thickness measurements were taken with calipers of 1/10mm precision and the average thickness was 4.85mm, The measurements were within 0.1mm of each other. When this exact same cord was under load in the cheiroballistra torsion spring, tuned to 250hz in one particular stretcher, its diameter had been reduced to 4.25mm, i.e. by ~12.3%. That said, there was lots of easy stretch in the rope, as can be seen from these pictures:
The cord could be very easily stretched about 5% from ~10cm to ~10.5cm. At 10.5cm the resistance increased dramatically, and this must be the point where the "slack" is mostly gone. This is in line with the tests conducted by Clift et al (2004: 59), where they noticed the reduction in the diameter of the rope was rapid at first, but slowed down progressively, especially with handmade ropes.
In any case, the musical method of plucking the cord and listening to the sound pitch is in my opinion much more useful than measuring the diameter of the cord; this is because it allows defining the return speed of the cord, which is independent of the rope diameter.