Applying computer vision for control the school of fish robots (SFR) using swarm model is a new direction. In this paper, the computer vision was studied for control a SFR using swarm model. The computer vision will return the framework and an algorithm was used to take the real position for every member. In the swarm model, every member has to solve a problem and combination together to get the target of the swarm. Combination is a complex problem, because the school want to solve a problem together, they have to combine the velocity, acceleration, force, direction. Moreover, the swarm fish robot will using computer vision to solve a problem, and the proposed problem is moved the chunk of wood (CW) on the straight line from point to point. At every time, to depend on CW position in the framework, every member will have suitable actions to get the target. Every member has to swim combination of direction, velocity, acceleration and force together to give a total of forces on the CW, and this total of forces have a straight direction to take the CW flowing on the straight line. With the proposed theory and algorithms, the scholar have designed a school of three fish to research and confirmed by experimental results.