From the looks of your two pics, WyDiablo, it looks like you have your connections like this:
1) Camera power separate from VTX power. First, is this necessary? It looks like both batteries are the same number of cells. Are they different type resulting in different voltage? If, and only if, they are the same type and cell count, consider only using one and connecting both camera and vtx power wires to the leads on the battery. In either case, by running two separate batteries powering the VTX and camera individually you are going to get an unstable ground reference between the two devices and it will impact your video quality (since the video is all that connects them). You should connect the grounds/negatives on the two batteries together such that camera black, vtx power black, and the two negatives on the batteries are all connected together. Based on your picture in
Need your advice guys - Page 2 you can accomplish this by running an additional wire connecting the black of one battery to the other battery black.
2) VTX AudioL (orange) connected to Camera yellow (video). You keep saying that the vtx red wire is the video input wire. Why? The VTX looks like a 5 pin plug and both the two end pins are unused on your plug, leaving the three middle pins with pigtail ends. According to the labeling on the VTX the three are: pin 2 (orange) is audioL in and should be left unconnected in your use case, next down (red) is video in, next after that (brown/black) is gnd (hopefully can also be left unconnected in your case, only needed if you're sourcing 5V to a camera through that connector, we're handling common ground by connecting the black battery wires as described in #1, might need to tie to the battery ground if video is bad, as Lee indicated). You need to connect camera yellow (video) to video input on the vtx, which I humbly submit is the middle wire, red. If we are wrong the worst that will happen is you connect the camera video output to ground, which isn't going to hurt anything.
I think these two changes will make this work, assuming the camera works.
You said you have a different camera which you've tried and it works. Did you connect it with the same VTX connector shown in the picture? Or do you have a separate plug on the camera which plugs into the VTX board?