1

Manufacturing E-Quality Through Integrated Web-enabled Computer Vision and Robotics

Richard Chiou* and Prathaban Mookiah

Applied Engineering Technology

Drexel University 3001 Market St. Suite 100

Philadelphia, PA 19104, USA

Yongjin Kwon Industrial and Information Systems Engineering

College of Engineering Ajou University, South Korea

*Author to whom correspondence should be addressed. Tel: +1 215 895 5960; Fax: +1 215 895 4988; E-Mail: ryc23@drexel.edu Abstract This paper investigates the development of a Web-based quality control system with the use of Internet-based computer machine vision. Such a system allows designers located far from the production facility to monitor, control, and program the quality inspection process as the product design evolves. Quality control tasks are performed and monitored remotely through the Web where Internet-based machine vision sensors inspect part dimensions and tolerance while Internet-enabled robots perform the necessary robotic operations. The remote quality monitoring system—which lets the user monitor the part status including data, image, and video through the Internet—is built using Java. The use of Java allows for interconnection and data transfer between the different components of the Web-based quality system and the robots. The ability to access and control quality control devices via the Web benefits the current manufacturing environment in the following ways: ubiquitous access to the production floor, remote control/programming/monitoring capability, and integration of production equipment into information networks for improving efficiency and product quality. Keywords: Quality – Web – Internet – Machine Vision– Robot – Manufacturing

1. Introduction

The vision for tomorrow’s factory is to integrate design, manufacturing, quality, and business with the Internet. It is a trend that Web-based gauging, measurement, inspection, diagnostic system, and quality control have become critical issues in the integration with e-manufacturing system and information management. Ever since its inception a few decades ago, the Internet has become a powerful tool in the areas of human activity including commerce, manufacturing, medicine, and education. Different equipment is now being connected to the Internet for a myriad of remote access

2

applications. One such area, e-manufacturing, integrates every level involved in the manufacturing process from the factory floor—suppliers to all the way up to the customers—through the Internet medium [1-6]. A prime impetus behind the penetration of the Internet into the manufacturing sector is to improve efficiency, subsequently increasing productivity and reducing the time it takes to deploy the product to the market. Different activities ranging from product design, prototyping, task scheduling, preventive maintenance and troubleshooting support, and machining and assembly [7-12], which have established norms in the manufacturing industry, are now seeing a sweeping change in the way they are being done with the introduction of e-manufacturing techniques. As a relative newcomer to the manufacturing landscape, the field of Internet robotics has opened up an immense number of applications [13-15]. Machine vision technologies have also kept pace with the development. High performance machine vision systems have enabled industries to perform product quality control at high production efficiency [16-18].

The constituents include highly advanced form of production equipment and

sensor networks connected to the Internet, all of which identified by the unique Internet Protocol (IP) addresses. The remote accessibility and the ability to control the equipment over the Internet present unprecedented benefits to the current manufacturing environment. Designers located remotely from the production facility can carry out the inspection and quality inspection as their design processes evolve [19]. The quality control issues and the positioning tolerance analysis of equipment can be tested according to the assembly and manufacturing specifications. Any changes in the product specifications and associated quality control routines can be instantly updated and verified, which will enhance the overall production efficiency. In the event of defective products, the operators are warned immediately through telecommunications networks, such as text messages, email, and phone calls. Despite the perceivable benefits and enormous potential for improving manufacturing operations, the Internet based manufacturing systems complemented by the remote quality control schemes have not appeared in the integrated format [20-21]. With the possibility of such real-time inspection, remotely located designers can constantly monitor and update the quality control routines and perform corrective actions. It eliminates the need for a human operator to be present at the floor and allows for collaborative work among a group of people. It provides increased flexibility to pull up relevant historical data and compare performance as well as storage of current observation.

In this paper, a remote quality diagnosis system is developed to monitor part quality status through a Web-based machine vision integrated with a robotics system. The system uses Java to integrate monitoring, diagnosis, and control-related data. The fundamental functionality of this system is to provide users who are located in remote locations with a live analysis on the quality of the products that pass on the conveyor belt in the production line. A machine vision system can be used to make non-contact measurements on the key dimensions and position of the parts and detect their quality. The measurements are fed back to a Web-based application server, which in turn distributes this information to remote users connected to it through a Web browser. Based

3

on the measurements and position information, the robot is directed for assembly operations according to its quality strategy. 2. System Development

Web-based quality control systems can provide remote sensing, monitoring, and online quality diagnosis for robotics and automation in manufacturing processes. The quality control issues and the tolerance analysis can be completed according to the assembly and manufacturing specifications. Any changes in the product specifications and associated quality control routines can be instantly updated and verified, which will enhance the overall production efficiency. Figure 1 shows a workstation used for the Web-based quality control. The experimental setup includes the following items: Yamaha SCARA robot YK-250X, RCX40 robot controller with optional on-board Ethernet card, Cognex DVT 540 machine vision, and HP m1050e PCs. The RCX40 controller is connected to the Ethernet and controlled using a PC/Server. Two DLink DCS-5300 Web cameras are used for constantly viewing the robot movement. These devices are connected by Internet Protocol (IP) address over the network. The word network refers to the connection between the devices in the work area as well as with the users. Network used here is local area network (LAN), which is connected to Web servers. Each device connected to the network has a unique IP address, which is used to connect to the others devices in the network. The IP address is separated into network address and host address sections. The network address section is extracted from the IP address by AND processing with the subnet mask.

3. Architecture of the Remote Quality Control System

Figure 2 shows the architecture of the system setup used for the study. The system

uses a Cognex DVT 540 computer vision system for inspection and measurement [22]. It is self-contained with optics and illumination LEDs to form an image of the part; an image acquisition circuit board to transform the image information into a computer format; and specialized image processing software to inspect parts, make measurements, and extract spatial feature information. The image resolution is 640 x 480 pixels with an exposure time of 4 ms and a frequency of 2 Hz for taking snapshots on a moving conveyor. It is equipped with an Ethernet port, which allows it to be accessed using an IP address and TCP (Transmission Control Protocol) port for information exchange. The extracted measurements are reported back to the application server through a TCP/IP connection established through this port. The in-built scripting support tools are also used to establish and control outside communications with the camera.

As with other manufacturing processes, robotic assembly operations require an

integrated quality control routine in each stage of the assembly. This ensures that no defective parts will be moved to the downstream, hence reducing the manufacturing costs associated with non-compliant quality. The Yamaha YK-250X SCARA (selective compliance assembly robot arm) robot is specifically configured to have a high rigidity along the vertical axis, while compliant along horizontal directions in the form of swing arm motions [23]. This renders the robot particularly suitable for pick and place or

4

assembly operations with a high degree of accuracy and speed, having the repeatability along horizontal planes of +/- 0.01 mm (+/- 0.0004 in.). A variable speed Dorner 6100 conveyor system is connected with the robot’s I/O device ports in order to synchronize the conveyor with the motion of the robot. The robot’s RCX 40 controller is equipped with an Ethernet card and runs a Telnet daemon. This enables exchange of textual commands based on the operation of robot over the Internet. The communications protocol utilizes TCP/IP. Once the connection is established, commands (e.g., speed, read and load program, turn motor on and off, and move to points) can be sent to the controller to achieve desired results. The robotic arm is composed of two links that are coupled with one another only at one end. The resulting movement is in x-y horizontal plane. The end-effecter attached to the second link moves in the vertical z-axis also provides for rotational r-axis. Thus the robot’s movement can be mapped by a Cartesian coordinate system. A point in the robot coordinate space can be represented as (ai, bi), where i denotes the point’s subscript. These conventions, along with selected calibration points and the vision axes, are shown in Figure 3. The image coordinates (xi, yi) are mapped into robot coordinates (ai, bi) using a coordinate transformation.

Considering the work area as a 2D surface, the scaling factors for the image X and Y axes are given as:

( ) 1 ( ) 11 0 2 0[ ] [ ] ; x y

x yS I x x S I y y− −= ⋅ − = ⋅ − (1) where Ix = the width of image (mm) at a particular focal length, Iy = the height of image (mm) at the particular focal length, and x0 & y0 = the pixel coordinate zero point. The point pairs, (xi, yi) and (ai, bi), represent the same point denoted by image and robot coordinates, respectively. The obtained scaling factors can be used to map the position and dimensions of a part from the pixel values to robot metrics through Equation 2 for any arbitrary point, denoted as (x, y):

( ) ( )0 0 ; x y

x ya a S x b b S y= + ⋅ = + ⋅ (2) Equation (2) can be represented as

R = T . I + R0 (3)

where the robot co-ordinate vector R = ⎥⎦

⎤⎢⎣

⎡

x

x

ba

, scaling matrix T = ⎥⎦

⎤⎢⎣

⎡)(

)(

00

y

x

SS

, image

coordinate vector I = ⎥⎦

⎤⎢⎣

⎡yx

, and the reference robot coordinate vector R0 = ⎥⎦

⎤⎢⎣

⎡

0

0

ba

. Since

the detected part’s position can be mapped to robot coordinates, it can be subsequently used to guide the robot to a correct pickup position. 4. System Software

5

At the core of this system is the Java-based software that integrates all the hardware components and provides the user with a unified view of the system in terms of quality control operation and information exchange. The software is implemented as three separate entities: scripts on the vision sensor, the application server, and the Applet based client interface. Figure 4 shows the overall software structure implemented in the Web-based quality control system.

The measurements made by the machine vision sensor are formatted and transferred to the application server through two scripts. Inter-script communication between these two scripts happens by setting and clearing status flags in the common memory registers. The first script, called the inspection script, is executed after each snapshot. The second script, called the main script, runs constantly in the background. It is responsible for establishing and maintaining communications with the application server. It uses the standard Java style function Socket to open a TCP connection to the application server on a dedicated port. Once the connection is established, it exchanges information on this connection with the application server through the functions send and recv.

The application server is a communications program written in Java. The operations of the program are divided and implemented through three main Java classes and a main program in accordance with the object-oriented programming model. The main program is the part of the code that is executed first when the application server is started. The main program is shown below, public class Main { public static void main(String[ ] args) throws Exception {

………………. ……………….

ServerSocket BrowserListenSocket = new ServerSocket(90); System.out.println("Waiting for connection from browser"); new Thread(new Operations(new BrowserSession(BrowserListenSocket.accept()))).start(); System.out.println("Thread Started");

………………. ……………….

} }

The main program starts by waiting for a TCP connection on a predetermined port from a browser through the ServerSocket class object. Since the end user has to activate the inspection process, this is deemed as the appropriate entry point into the program. This connection is initiated by the browser through a Java Applet, which is described later. Once a connection is accepted through the accept method, an object of BrowserSession class, which is a class designed to handle all the browser communication aspects is initialized with this new connection. This newly initialized object is then used as a parameter to initialize an object of the Operations class. The Operations class is designed to handle the communications with the machine vision camera and the robot.

6

The Operations object implements the Java Runnable system class—therefore being able to run as a thread. A thread as a mini-program within a program can execute in parallel with other threads. The reason behind assigning the browser communications to a separate class and the robot-camera operations to another class should be evident from the way the system is organized. The browser forms the client part of the operations while the robot and camera constitute the server side of the system and hence this division. Once the initialization of the objects is completed, a new thread is started with the newly initialized Operations object and the thread is started. This effectively establishes the required connections between the entities in the system and starts operation.

The scenario of Figure 5 is an example of using the main program as a transfer function between the vision data acquisition system and application server, correlating data and transforming them to Web server. These data can be gathered from traditional control I/O or through a separate data acquisition systems using different communication protocols. The decision logic for quality control in the application server is then used to process real-time data from the database where all measurement data are stored. The data are also packaged for visualization on the Web. Once a decision is made, it instructs the robot to perform the necessary action on the inspected part. The end user interface is implemented on a Web browser through a Java Applet. Information is exchanged between the Applet and the application server through a TCP/IP connection. The graphical user interface is implemented through the use of Java swing components. This Applet is served through an Apache Web server. This sequence of connection establishment between the Applet and the application is shown in Figure 5. During the first step, shown in Figure 5(a), the user requests the Web site containing the Applet from the Web server by sending a HTTP request. The Web server responds to the request by sending the HTML page. Once the Applet is downloaded, compiled, and run on the client machine, the second step occurs. During the second stop, shown in Figure 5(b), it establishes a connection with the application server directly and starts exchanging data.

Figure 6 shows a screen capture of the Web-based end-user interface. The Applet does not execute any task upon loading. The user initiates the inspection procedure by clicking on the “Start Inspection” button. Once clicked, it attempts to establish a connection with the application server. Successful establishment of the connection also starts the inspection process at the other end, as explained in earlier sections. There are several fields in the Applet that provides a live analysis of the ongoing inspection process such as the number of parts inspected, number of compliant parts, current dimension, and cumulative measurement history.

The browser interface also includes two Web-camera views and a machine vision view. The Web-cameras are equipped with an embedded Web server which captures stream of images over the Internet using the HTTP protocol. The Web interface incorporates Microsoft ActiveX objects provided by the manufacturers that allow users to directly embed these views in the Web site with minimal configuration. It also eliminates the need of any specific software in the user’s end—a Web browser takes care of it. Since the programs, data, and access information are stored centrally, changes are made

7

available to users immediately. The current Web technology also provides for extensive interactivity allowing for real-time interaction with the system. 5. A Case Study for Web-enabled Robotic Control

The system was studied using the test parts whose geometry is shown in Figure 7.

Six key dimensions: length (L), width (W), diameters of the two circles (D1, D2), horizontal center-center distance between the circles (CCL), and vertical center-to-center distance between the circles (CCW) are shown in the figure. These parts were machined with a tolerance limit of +/- 0.25 mm. Parts whose dimensions lay outside this range were rejected. Several parts were made, with a few purposely machined out of tolerance limits on each dimension. Several parts from each type were mixed up and fed through the conveyor belt for inspection. The specifications of the different test parts used for testing are shown in Table 1. Although this part does not pose any serious measurement challenge, it presents a moderate complexity for the requirement of the work. The part thickness was limited to less than 1 mm, since a higher thickness would result in problems due to shadows. Shadows would trick the camera to see the part as being elongated on some sides and lead to wrong measurements. 6. Internet Based Machine Vision System

Machine vision packages for the Cognex DVT 540 computer vision system are configured as a set of tools for inspections and measurements [24]. SoftSensors are the working class inside Smart Image Sensors. Every type of SoftSensor serves a specific purpose, and the combination of the SoftSensor results represents the overall result of the inspection. The main groups of SoftSensors include Presence/Absence SoftSensors, Positioning SoftSensors, and Specific Inspection SoftSensors. The SoftSensors help locate parts to be inspected. In most applications, the type of positioning SoftSensor becomes the key to a successful setup because they locate the part to be inspected and pass an internal reference to the remaining SoftSensors indicating the location of the part. The SoftSensors perform basic checks for detection of features and edges or analyze the intensity level of the pixels in a certain area or linear path. All the SoftSensors perform many tasks from very specific inspections to a user defined task (programmable). They include: Measurement, Math Tools, Readers, Blob Tools, Template Match, ObjectFind, Pixel Counting, Segmentation, SmartLink, Script, and Spectrograph. The script is a programmable tool that can access other SoftSensors for data gathering, etc.

The vision system is set with the following image parameters: CCD exposure time: 2 ms; digitizing time: 26 ms; image processing and inspection time: 10 ms; and the time from image capture to inspection finish: 39 ms. The intensity of ambient lighting is enough for the camera CCD sensor (with the use of integrated white LEDs), so the exposure time of 2 ms was found to be adequate. The part is carried by the conveyor with a speed of 125 mm/sec in accordance with the pre-defined production rate. The part then takes 680 ms to pass through the inspection area set by the current camera field of view. The maximum number of inspection within 680 ms is 17 times with the current image parameter setting, meaning that many inspections can be performed on the same part.

8

Since the vision system only sends out a very small string of text data, there was no delay problems associated with the large data transfer over the Internet in manufacturing processes.

The machine vision camera is initially trained to learn the profile and make measurements of a part being tested through the FrameWork software. Each inspection and measurement task is defined through several soft-sensors. Figure 8 shows five soft-sensors defined for detecting the part, and extracting its lengths, widths, radii, and center-to-center distances. The “WorkPiece” is an ObjectFinder soft-sensor defined by the rectangular box as shown in the figure and trained to inspect the shape and position of the product when the product arrives inside the area covered by the image box. The rest of the soft-sensors make the measurements corresponding to their names. Once the part is detected by the ObjectFinder soft-sensor, the measurements are made automatically by the camera through the measurement soft-sensors. Although the camera can be set to detect and measure the parts, the task of formatting the measurements and communicating it to the server is a custom job that is completed through the in-built Java-based scripting tool.

The camera extracts the first four features and the center-to-center distance between the circles. The center-to-center distance was measured instead of the last two quantities, CCW and CCL, since they can be jointly quantified by the center-to-center distance CCD. As an example, the distance CCD is used for testing Web-based statistical process control. This, however, requires us to develop a tolerance limit based on which the part feature has to be classified. The tolerance limit for CCD determined using the root sum of squares (RSS) approach [25]. CCD, CCW, and CCL have a simple geometrical relationship given by:

22LWD CCCCCC += (4)

Since this is a non-linear relationship, the non-linear RSS extension is employed. The variance of CCD is given by:

)()()()(

)()()(

22

LL

DW

W

DD CCVar

CCCCCCVar

CCCCCCVar ⎟⎟

⎠

⎞⎜⎜⎝

⎛∂∂

+⎟⎟⎠

⎞⎜⎜⎝

⎛∂∂

= (5)

where ∂ is the partial derivative operator and Var(.) is the variance. For quality control, the standard deviation would correspond to the full tolerance limit and hence, the following relationship holds:

Var(CCK) = Tol(CCK) (6) where K corresponds to either D,W or L and Tol(.) is the corresponding tolerance limit. Hence Equation (5) can be written as:

9

)()()()(

)()()( 2

22

2

LL

DW

W

DD CCTol

CCCCCCTol

CCCCCCTol ⎟⎟

⎠

⎞⎜⎜⎝

⎛∂∂

+⎟⎟⎠

⎞⎜⎜⎝

⎛∂∂

= (7)

Differentiation of Equation (5) with respect to CCL and CCW yields;

(8)

22)()(

LW

W

W

D

CCCCCC

CCCC

+

−=

∂∂ (9)

Substituting Equations (8) and (9) into Equation (7) results in

)()()( 222

22

22

2

LLW

LW

LW

WD CCTol

CCCCCC

CCTolCCCC

CCCCTol

++

+= (10)

The following numerical values are substituted from the design specifications:

CCW = 26 mm, CCL = 26 mm, Tol(CCW) = ±0.25 mm, and Tol(CCL) = ±0.25 mm. It results in the same tolerance limit of ±0.25 mm for CCD. The process control chart corresponding to the measured center-to-center distances for a trial run is given in Figure 9. The CL line corresponds to the mean value which is calculated to be 36.95 mm. Therefore the upper control limit (UCL) = 37.05 mm and lower control limit (LCL) = 36.55 mm corresponding to the mean value with a tolerance of ±0.25 mm. The four offshoots that lie outside these limits correspond to the non-compliant parts (types 6 & 7) that were intentionally mixed up with the compliant ones. Otherwise it can be seen that the center-to-center distance measurements follow the expected statistical behavior with a tolerance of ±0.25 mm as calculated above. The plots show that measured values closely follow the actual values but almost always take a slightly higher value. This can be attributed to the weak shadows at the edges along with the non-linear effect s of the lens, which results in a slightly expanded image of the part which in turn results in an overestimation of the dimensions under the camera’s view.

Upon starting the browser, the Java Applet starts running and creates the control interface, which is displayed to the user. When the “Start Inspection” button is clicked, a connection is made to an intermediate application server, which is listening for incoming connections. Once an acknowledgement is received from the application server, it goes back on the detection mode and waits for the next part. The flow chart describing the entire sequence of operation is shown in Figure 10. This server then creates a new thread for this particular Applet instance and connects to the camera and the robot. The

22)()(

LW

L

L

D

CCCC

CCCCCC

+

−=

∂∂

10

application then starts the conveyor and waits for part detect event. When the parts are detected by the camera, the conveyor is stopped, and the measurements are recorded in the registers in the framework. From there, they are passed to the server.

Figure 11 shows Web-enabled robotic operations for picking and placing a part in either stack A or B, depending upon compliant or non-compliant parts detected by the Internet-based machine vision system. The server then calculates whether this is a compliant part or not by checking the dimensions of the part compared to the pre-defined dimensions. If it is compliant, then it is stacked with the other compliant parts; otherwise, it is kept in the stack compliant. The count is then increased at the Applet end, which lets the user know the status of compliant versus noncompliant parts encountered. The conveyor is then restarted, and this process continues until the user logs out. The following is a part of the program that checks the dimension and stacks into two groups via the Internet. The first two lines check the orientation of the part on the conveyor, and the rest of it deals with checking the dimensions and placing it in two stacks.

if (Angle > 90) Angle = Angle - 360; if (Math.round(CenCenM) < 36.55|| Math.round(CenCenM) > 37.05) { ThisSession.WriteMessage("Non Compliant Part\n"); OutputMessage(RobotConnection.sendCommand("@MOVE P, P126"));

………………. ……………….

} else { ThisSession.WriteMessage("Compliant Part\n"); OutputMessage(RobotConnection.sendCommand("@MOVE P, P127, P128"));

………………. ……………….

} 7. Conclusion

This work successfully demonstrates a concept of E-Quality for Manufacturing

(EQM) through the implementation of a Web-based quality control system. A computer vision system measures the dimensions of products on the conveyor belt and sends the information to an application server, which activates an appropriate action for the robot. Various image processing and analysis algorithms are integrated for remote quality inspection. Operators can remotely adjust the inspection routine in the case of process changes, such as lighting conditions, part variations, and quality criteria. This result provides a great impact in production since engineers can access and control the equipment and quality anytime, from anywhere. Acknowledgement

11

This work is supported by the National Science Foundation (Award # DUE-0618665, CCLI Phase II Project on EQM) and Yamaha Robotics Company. The authors wish to express sincere gratitude for their financial support. References 1. Jose, Joao and Ferreira, Pinto, ”E-Manufacturing: Business Paradigms and

Supporting Technologies,” Kluwer Academic Publishers, Dordrecht, The Netherlands. Pp. 280, 2004.

2. Kwon, Yongjin and Chiou, Richard, “Remote, Condition-based Maintenance for Web-enabled Robotic System,” To be published in Robotics and Computer Integrated Manufacturing, 2008.

3. Ming, X. G., Yan, J. Q., Lu, W. F., and Ma, D. Z., “Technology Solutions for Collaborative Product Lifecycle Management – Status Review and Future Trend,“ Concurrent Engineering, Vol. 13, pp. 311 – 319, Dec. 2005.

4. Lacroix, Eric and St-Denis, Richard, “Web Technologies in Support of Virtual Manufacturing Environments,” Proc. Conf. on Emerging Technologies and Factory Automation, Lisbon, Portugal, Vol. 2, pp. 43- 49, Sept. 2003.

5. Lee, Jay, “E-manufacturing—fundamental, tools, and transformation”, Robotics & Computer-Integrated Manufacturing, Vol. 19, No. 6, pp. 501 – 508, Dec. 2003.

6. Ali, Ahad, Chen, Zaifeng, and Lee, Jay, “Web-enabled platform for distributed and dynamic decision-making systems,” Published online, July 2007.

7. Wang, Lihui, Orban, Peter, Cunningham, Andrew, and Lang, Sherman, “Remote Real-time CNC machining for Web-based Manufacturing,” Robotics & Computer-Integrated Manufacting, Vol. 20, No. 6, pp. 563-571, Dec. 2004.

8. Ahn, Sung-Hoon, Kim, Dong-Soo, Chu Won-Shik, and Jun, Cha-Soo, “MIMS: Web-based micro machining service,” Int. J. Computer Integrated Manufacturing, Vol. 18, No. 4, 251 – 259, June 2005.

9. Zhou, Guanghui, Jiang, Pingyu, and Fukuda, Shuichi, “Using Mobile Agents to Schedule a Manufacturing Chain on the Internet,” Concurrent Engineering, Vol. 10, pp. 311-323, Dec. 2002.

10. Shen, Yantao, Xi, Ning, Lai, King W.C. and Li, Wen J., “Internet-based Remote Assembly of Micro-electro-mechanical systems (MEMS),” Assembly Automation, Vol. 24, No. 3, pp. 289-296, 2004.

11. Cachero, Cristina, Calero, Coral, and Poels, Geert, “Metamodeling the Quality of the Web Development Process’ Intermediate Artifacts,” Journal of Advanced Manufacturing Technology, pp. 74–89, 2007.

12. Jiang, Pingyu and Zhang, Yingfeng, “Visualized Part Manufacturing via an Online e-Service Platform on Web,” Concurrent Engineering, Vol. 10: pp. 267-277, Dec. 2002.

13. Wang, Lihui, Xi, Fengfeng and Zhang, Dan, “A Parallel Robotic Attachment and Its Remote Manipulation,” Robotics and Computer-Integrated Manufacturing,” Vol. 22, Issues 5-6, pp. 515-525, Oct-Dec 2006.

14. Wang, Lihui, Sams, Ryan, Verner, Marcel and Xi, Fengfeng, “Integrating Java 3D Model and Sensor Data for Remote Monitoring and Control,” Robotics and Computer-Integrated Manufacturing, Vol. 19, Issues 1-2, pp. 13-19, Feb-Apr 2003.

12

15. Yang, Yu, Zhang, Xiaodong, Liu, Fei and Xie, Qiu, ”An Internet-based Product Customization System for CIM,” Robotics and Computer-Integrated Manufacturing, Vol. 21, Issue 2, pp. 109-118, April 2005.

16. Xu, Peiquan, Xu, Guoxiang, Tang, Xinhua, Yao, Shun, ”A visual seam tracking system for robotic arc welding,” Journal of Advanced Manufacturing Technology, 10 January 2007.

17. Horst, R.L. and Negin, M., “Vision System for High-resolution Dimensional Measurements and On-line SPC: Web Process Application,” IEEE Transactions on Industry Applications, Vol. 28, Issue 4, pp. 993 – 997, July-Aug 1992.

18. Michael, Lang and Fitzgerald, Brian, “Web-based systems design: a study of contemporary practices and an explanatory framework based on ‘‘method-in-action’’,” Published online Journal of Advanced Manufacturing Technology, July 2007.

19. Lee, G., Mou, J. and Shen, Y., ”An Analytical Assessment of Measurement Uncertainty in Precision Inspection and Machine Calibration,” International Journal of Machine Tools and Manufacture, 37 (3), pp. 263–276, 1997.

20. Kwon, Y., Rauniar, S., Chiou, R. & Sosa, H., “Remote Control of Quality Using Ethernet Vision and Web-enabled Robotic System” Journal of Concurrent Engineering: Research and Applications, Vol. 14, No. 1, pp. 35-42, 2006.

21. Kwon, Y., Chiou, R., Rauniar, S. & Sosa, H., 2006, “Positioning Accuracy Characterization of Precision Micro Robot over the Internet,” Journal of Advanced Manufacturing Systems, Vol. 5, No.1, pp. 45-57, 2006.

22. Cognex Smart Image Sensor DVT 545 Series Manual. 23. Yamaha Robot Operations Manual RCX40. 24. Cognex Smart Image Sensor FrameWork Manual. 25. Creveling, C. M., “Tolerance Design – A Handbook for Developing Optimal

Specifications,” Addison-Wesley, 1997.

13

Figure 1. A workstation with a Yamaha robot, machine vision, controller, conveyor, and mechanical devices for Web-based quality control.

Figure 2. Web-based vision & robotic system integrated with E-manufacturing.

LEDs

Network Cameras

Ethernet-based Machine Vision

Conveyor

Web-cam

YK-250X SCARA Robot

Application Server

Application Server

Users

Internet

Vision Camera

Robot

Part Image

Quality Information

Robotic Operation

E-manufacturing system

14

Figure 3. Tracking a moving part over the Internet.

Figure 4. Software structure

CLIENT SIDE

Web Browser

Quality Information

MFC Library for ActiveX Controls

SERVER SIDE

Apache Web Server

Client Interface Java Applet

Application Server (Java)

Robot Communications Module

Inspection Camera Communication Module

Client Communication Module

Main Control & Decision Logic

ROBOT CONTROLLER

Telnet Daemon

INSPECTION CAMERA

TCP Sockets

TCP/IP Socket Communication

Java IPC

TCP/IP Socket Communication

HTTP

Java IPC

Java IPCWeb CAMERA

HTTP Server

Image Transfer

Telnet Protocol

(x0, y0) (x0, y0)

15

(a)

(b)

Figure 5. Sequence of connection establishment between client and server

Figure 6. Web interface for the quality measurement and robotic operations

Applet Server

Web Server

Internet TCP

TCP

User (Browser Running Applet)

Application Sever

Applet Server

Web Server

Internet HTTP Request

HTTP Request

HTTP Response containing Applet

HTTP Response containing Applet User

(Browser)

Application Sever

Object Finder Soft Sensor Trained to

Detect the Part

Quality MeasurementsRobotic

Operations

16

Figure 7: Test part geometry

Figure 8. Configuration of machine vision for measurements

Length

Width

Radius 1

Center - Center Radius 2

WorkPiece (‘ObjectFinder’)

θ

CCw

CCL

L

W

�D2

�D1

17

Figure 9. Statistical process control chart for center-to-center CCD

Figure 10. Flow chart for machine inspection procedure via Internet

0 5 10 15 20 2536.5

37

37.5

38

38.5

39

Sample Number

Mea

sure

d D

ista

nce

(mm

)

MeasurementsCLUCLLCL

On click ”Start Inspection” Connect to Application

Start part Detected?

Inform Application Server and

Check Part Quality

Non-Compliant

Yes Robotic Operations

Compliant

Stack A

Stack B

Next Part

Part rejected

UCL

LCL

CL

18

(a) (b) (c) Figure 11. Web-enabled robotic operations for: (a) picking up a part on the conveyor, (b) moving the part, and (c) placing the part on stack A (compliant).

Table 1: Workpiece Specifications (All units are in mm except which is in degrees) Type L W D1 D2 CCL CCW θ Remarks

1 50 50 12 12 26 26 45 Correct Dimensions

2 52 50 12 12 26 26 45 Wrong Length 3 50 52 12 12 26 26 45 Wrong Width 4 50 50 14 12 26 26 45 Wrong Radius 5 50 50 12 14 26 26 45 Wrong Radius 6 50 50 12 12 28 26 45 Wrong Center – to

- Center Horizontal Distance

7 50 50 12 12 26 28 45 Wrong Center – to - Center Vertical Distance

8 50 50 12 12 26 26 47 Wrong Slope

Stack A

Stack B

Object Object Object