The Effects of Using Some Different Types of Learning Automata on Presentation of Adaptive Neighbor-Based and Energy Efficient Topology Control Protocols in Wireless Sensor Networks

The high number of nodes and dynamic and periodic topological changes, as well as constraints in the physical size of nodes, energy resources, and power of processing are some characteristics of sensor networks that make them different from other networks. One method to overcome these constraints is topology control with the aim of reducing energy consumption and increasing the network’s capacity, which has the most influence on the network’s efficiency, especially in terms of energy consumption and lifetime. In consideration of learning Automata’s abilities, such as low computational load and adaptability to changes via low environmental feedbacks, in this paper, neighbor-based topology control protocols based on learning Automata have been proposed somehow that all nodes are equipped with Automata. The nodes try to adapt their selected actions with required conditions for creating a connected and energy efficient network by selecting the best radio range for themselves. This approach finally forms a proper topology, and in this way it lowers the network’s energy consumption in its lifetime. The exclusive characteristic of these methods is the high number of transmission ranges that each node can select as transmission radius. In the first proposed protocol, a P-model environment is used for learning phase, but in the second proposed protocol, a Q-model environment is applied. Simulation results show favorite functionality of proposed protocols in comparison with some other similar protocols from topology control point of view, as well as high improvement of achieved results for the Q-model environment. Keywords— Adaptive protocol; Energy Efficient; Learning Automata; Neighbour-Based; Topology Control; Wireless Sensor Networks.


I. INTRODUCTION
Wireless sensor networks are sets of large numbers of sensor nodes that are close to each other and scattered in the environment, and each of them, autonomously and in cooperation with other nodes, tries to achieve a special goal.Every node in this collection can communicate with others and gives its information to other nodes until finally reporting the control environment's status to a central point, the sink node.The principal aim of wireless sensor networks is monitoring and controlling variations of climate and physical or chemical changes of an environment within a deterministic region [1,2].Since nodes work autonomously and without human interference; and physically are very small with some constraints in power * Corresponding author.
processing, memory capacity and power resources and etc; so one of the goals for these networks is to reduce the energy consumption in order to prolong the network's lifetime.
Of the diverse methods that can be used for decreasing the consumed energy in a sensor network, selecting a suitable topology has the most influence on efficiency of the network in terms of energy consumption and lifetime.
Topology control in sensor networks is the art of coordinating nodes' decisions regarding their transmitting ranges, in order to generate a network with the desired properties(e.g.connectivity) while reducing node energy consumption and/or increasing network capacity [3].Several topology control protocols have been presented until now, in order to allow nodes to select and set their transmission ranges.These selections change based on priorities and different conditions such as less energy consumption, the network's sparseness, the lower node's degree, fault tolerance and decreasing interference [4].
With consideration of these mentioned constraints for sensors, the aim of much research has been focused on presenting approaches that, with simple control methods and with low cost, in addition to responding to the requirements, can stand for some constraints such as bandwidth, limited energy, environmental interferences, etc., and can correspond with general conditions based on the requirements and available wills, such as transferring high density rich in content information, long lifetime, low cost, etc.
Investigations on Learning Automata (LA) and wireless sensor network characteristics have shown that Learning Automata, with properties such as low computing overload, adaptability to distributed environments with ambiguous information, and adaptability to environmental changes, is a very suitable model for use in sensor networks [5].
Particularly because of energy restriction in sensor nodes and their requirement to reduce transmission of redundant information to prevent energy waste, wireless sensor networks' tendency to use such algorithms that can work in a distributed manner with local information has high importance.Because of this, in this paper neighbour based topology control protocols based on learning Automata have been proposed.In the first part LMNALA i is proposed, in which learning automata has been given to every sensor node that, with time, use, and consideration to conditions, will select the best transmission range for itself.In the second part another neighbour based topology control protocol based on learning Automata called LMNALAQ ii is proposed, which takes into consideration the learning automata's environment as a Q-model instead of a P-model in order to investigate its effects.
In previous presented methods the number of transmission ranges that each node could have selected was limited, but in this proposed method some efforts have been made to give nodes more optional selections that are close to each other, which are known as transmission ranges.
The rest of this paper is organized as follows.
Section2 is an introduction of topology control and related works on it.Learning automata as a basic learning strategy that is being used inthe proposed methods will be briefly discussed in Section 3. In Section4 the problem statement is defined.In Sections 5 and 6 the first and second methods are proposed, respectively.Simulation results are shown in Section 7. Section 8 is the conclusion.

A. Review of Topology Control
The goals of topology control mechanisms are to dynamically change the nodes' transmitting range [6] and to control the radio power level of nodes to achieve a connected and, of course, optimum topology, so that we can maintain some properties of the communication graph while reducing the consumed energy by node's transceivers, and also so that we can control the energy consumption since transceivers of nodes are one of the primary sources of energy consumption in Wireless sensor networks.These mechanisms themselves, regarding the essence of the network and the information that each node can obtain, are divided into two categories: homogeneous and non homogeneous topology control [3,4].In homogeneous topology control, all nodes of the network use the same transmission range r, and the topology control problem reduces to the problem of determining the minimum value of transmission range r somehow that network properties like connectivity are satisfied [3,4]; however, this kind of topology is not suitable enough from points of efficiency, lifetime or strong connectivity.
In non homogeneous topology control, nodes are allowed to choose different transmitting ranges and independently choose their transmitting range in order to besides preserve local connectivity minimize interference.Depending on the type of information that is used to construct the topology, it is classified into three categories [3].The first of these is location-based approaches, that nodes know some accurate information about their positions and with the use of this information try to construct a proper topology for network.Second is direction-based approaches; in these it is assumed that nodes do not know their positions but can estimate the relative direction of their neighbours.Third is; neighbour-based techniques, through which nodes have access to a minimal amount of information regarding their neighbours, such as their I.D number.

B. Related Works on Neighbour-Based Topology Control
Kneigh iii and XTC iv protocols are two distributed topology control protocols from non homogeneous topology control protocols and neighbour based ones [7,8].The goal of K neigh protocol is to keep at least k nearer neighbours to each node.Every node increases its amount of transmission power until it can communicate with its k neighbours directly.
In XTC protocol each node orders its neighbours based on the concept of link quality and distance.Then it communicates with neighbours that cannot communicate with them through other nodes for which the cost of direct communication would be lower than that for indirect communication.
Other topology control protocols can refer to RAA_2L, in which every node selects one from two level transmission ranges R s or where (R w <R s ) [9].If a node with R w transmission range can communicate with R s transmission range neighbours (either directly or through a whisperer range neighbor), then the node selects the R w transmission range; otherwise it selects the R s transmission range.In RAA_3L transmission ranges extend to three levels, and every node selects one from three transmission ranges, R s or R w or R t , so that their relations are as follows:(R w <R t <R s ).The used mechanism for RAA_2L is performed for two transmission ranges, R s and R t .If a node selects R t transmission range, then again the used mechanism for RAA_2L protocol will be performed for two transmission ranges, R s and R t.
Also, in [10,11] a topology control protocol based on irregular cellular learning automata (CLATC) v has been proposed in which an irregular cellular learning automaton is mapped to network.But each node in the network has its own individual learning automata and, in cooperation with LAs of its neighbor nodes, tries to choose the most suitable transmission range from three available ranges for itself in consideration of other nodes' transmission ranges, which has better correspondence to network's conditions than other transmission range.

III. LEARNING AUTOMATA
Learning automata (L.A) is an abstract model [12,13,14] that randomly selects one action out of its finite set of actions and performs it on a random environment.The environment then evaluates the selected action and responds to the automata with a reinforcement signal.Based on the selected action and the received signal, the automata updates its internal state and selects its next action.Figure 1 depicts the relationship between an automata and its environment.Environment can be defined by the triple } represents the output set, and } ,..., , { is a set of penalty probabilities, where each element c i of c corresponds to one input of action i  .An environment in which  can take only binary values 0 or 1 is referred to as a P-model environment.A further generalization of the environment allows finite output sets with more than two elements that take values in the interval [0, 1].Such an environment is referred to as a Q-model.Finally, when the output of the environment is a continuous random variable that assumes values in the interval [0, 1] it is referred to as an S-model.Learning automata are classified into fixed-structure stochastic automata, and variable-structure stochastic automata.In the following study, we consider only variable-structure automata.
A variable-structure automaton is defined by the represents the action probability set, and finally represents the learning algorithm.This automaton operates as follows.
Based on the action probability set p, the automaton randomly selects an action i  and performs it on the environment.After receiving the environment's reinforcement signal, the automaton updates its action probability set based on equations (1) for favorable responses, and equations (2) for unfavorable ones. (1) In these two above equations, a and b are reward and penalty parameters, respectively.For a = b, the learning algorithm is called [15].

IV. PROBLEM STATEMENT
In this paper it is assumed that every sensor node can choose a proper and deterministic value from the transmission ranges between [R w ,R s ],that is, between the low power transmission radius that is termed the whisperer radius(R w ), and the high power transmission radius(R s ) which is termed the shouter radius.There is also a medium power transmission radius between [R w ,R s ] which is called the speaker radius(R t ), whose value is proportional to network density and is defined with it [16].What is obvious from above is that, the R w transmission radius is lower than the other two transmission radiuses and the R s transmission radius is higher than the two others, which means (R w <R t <R s ).The value of the R w transmission radius is equal to 0.8*R t , and the value of the R s transmission radius is equal to 1.25*R t .In this model we convert the transmission range into n different values; the difference of each value from its previous value is equal to the difference of each value from its next value and is called Ainc, or Adec, which is defined as follows: Ainc= Adec =)R s -R w ( /n=C*R t In this relation C is a constant coefficient and is equal to0.45/n.The considerable problem in this paper is selecting the most suitable transmission radius from the transmission range between [R w , R s ] for each node and conserve the required conditions for network connectivity.

V. THE PROPOSED METHOD (LMNALA)
This proposed control topology protocol that is neighbor based is performing based on learning automata; and has two Phases as described below: Learning and selecting the best radio range.

A. Learning Phase
At the beginning of this stage, a learning automata is assigned to each sensor node in the network.The radio ranges of all nodes, initially, are equal to each other: Rt, which is proportional to network density.Learning helps nodes to choose the most suitable radio range.Each Learning automata has three actions, which are α 1 , α 2 , α 3 , which are, respectively, described as follows: Increasing the radio range of each node with a constant value, decreasing the radio range of each node with a constant value, not changing the nodes' radio range.It is initially supposed that, the selection probability of every action is equal to each other action.In other words, it is according to the following relation (3), where m is the number of learning automata actions: At first, all nodes randomly and simultaneously select one of their own L.A actions and then, affecting the selected actions on nodes' radio ranges and updating them, they start broadcasting hello messages, each of which contains the I.D number of the node, to all surrounding sensor nodes.Then, in the next stage, based on the number of responses to nodes' sent signals(N ack ), which is called acknowledgment reward or penalty is given for a selected action and forms reinforcement signal(B i ) as follows.Actually, the acknowledgment reward indicates the number of one node's neighbour or the same degree of each node.The number of received acknowledgments by a hello message sender node or the same (N ack ), compares with the minimum threshold (T l ) and maximum threshold (T h ) values that have been achieved from [17] and is necessary to guarantee optimal connectivity.If the number of received acknowledgments is between the minimum and maximum threshold, the correspondent probability to the action that caused this effect increases, and the probabilities of other actions according to relation (1) decrease.But if the number of received acknowledgments are greater than the maximum threshold (T h ) value or lower than the minimum threshold (T l ) value, then the correspondent probability to the action that caused this effect decreases and probabilities of other L.A's actions change according to relation (2).
The environment response to the selected action of learning automata is calculated as follows: ( ) desired response undesired response 0 : 0 3 : 1 : 0 3 : The reward coefficient is supposed to be constant, but the penalty parameter is calculated in a time variant manner and is proportional to the difference of the number of achieved neighbours from the selectedradio range with thresholds described as follow

 
The initial value for b should be selected so that the penalty coefficient never exceeds a determined restriction.With supposing constraint in each period for coefficient in process of calculating penalty parameter like above, we can prove that: 0<b i (t)<1 The proposed method in reference [18] has been used for calculating the consumed energy for transmitting and receiving packets to neighbour nodes.This phase that is contributing to the selection of automata actions and updating radio range according to that selection will be repeated until one of the following conditions occurs: one of actions' probability is higher than threshold, or the learning phase is repeated k times, which indicates the maximum iterations of the learning algorithm.

B. Selecting the Best Radio Range Phase
As mentioned before, this phase is executed exactly like the previously proposed protocol in section 5 is executed.

VI. THE PROPOSED METHOD (LMNALAQ)
This proposed control topology protocol that is neighbor based and is performing based on learning automata like the proposed protocol in section 5, has two phases: Learning and selecting the best radio range as follows.

A. Learning Phase
At the beginning of this stage, a learning automaton is assigned to each sensor node in the network.The radio ranges of all nodes are initially equal to each other R t , which is proportional to network density.The duty of each learning automata is to help nodes to choose the most suitable radio range.Each Learning automata has three actions: α 1 , α 2 , and α 3 , which are described, respectively, as follows: Increasing the radio range of each node with a constant value, Decreasing the radio range of each node with a constant value, Not to change the nodes' radio range.It is initially supposed that the selection probability of every action is equal to each other action.In other words, it is according to the following relation (6), where m is the number of learning automata actions: At first, all nodes randomly and simultaneously select one of their own L.A actions and then with affecting those selected actions on nodes' radio range and updating them, start broadcasting hello messages, containing I.D number of nodes, to all surrounding sensor nodes.After passing a little time, every node achieves the number of responses to sent signals by itself that indicates the number of one node's neighbors(N ack ) or the same, one node's degree and based on it evaluates its selected action and dedicates reward or penalty to it.The reaction of the environment to the automata's action is as follows.The number of received acknowledgments by a hello message sender node (N ack ) is compared with the explained minimum threshold (T l ) and maximum threshold (T h ) values [17], which are necessary to guarantee optimal connectivity between nodes.
We define one relation for reward and two relations for penalty: b high and b low values in relation ( 7) and ( 8) are penalty parameters, and it is assumed that b high is higher than b low such that b high = 2 * b low .So, relation (7) in comparison with relation ( 8) penalizes automata's action with a higher value as a penalty coefficient.Reinforcement produced signal (B i ) can have one of the available values in the following triple set with three values {0, 0.5, 1}; thus, the environment's model is Q. if B i produces a value equal to 0, a reward is given for the performed action.If B i produces a value equal to 0.5 and 1, a penalty is dedicated to the performed action, so that 1 indicates a higher penalty coefficient and 0.5 expresses a lower penalty value as a coefficient.The reinforcement signal is calculated as follows: The learning phase and updating the radio range according to the selection made with the aid of learning automata continue until one of the actions' probabilities is higher than threshold, or until the number of learning iterations and updating the radio range reaches a defined threshold.
The proposed method in reference [18], the same as in the previous proposed protocol is used, for calculating the consumed energy for transmitting and receiving packets to neighbour nodes.( 7)

B. Selecting the Best Radio Range Phase
As mentioned before, this phase is executed exactly like the previously proposed protocol in section 5 is executed.

VII. EXPERIMENTAL RESULTS
To evaluate the performance CLATC, RAA_2L,RAA_3L protocols [9] and homogeneous case [16] have been simulated in the N.S2 simulator and the achieved results have been compared to each other and with the results from our proposed methods (LMNALA), (LMNALAQ) in the presented simulator in [19].In simulation, nodes have been distributed in a region of 1250*1250 square meters.The number of sensor nodes ranges from 200 to 600 nodes, and it is assumed that nodes have the same initial energy.Every node has a transmission range between [R w ,R s ].The medium transmission radius, which is proportional to network density, is assumed for 200 nodes equal to 109 meters, and for 300 nodes equal to 86 meter, for 400 nodes equal to 74 meter, for 500 nodes equal to 67 meter, for 600 nodes equal to 60 meter.The R w , R s transmissions radiuses are defined according to R t .The energy model that has been used in these simulations is what that is expressed in [18].Learning iterations for automata are limited to a maximum 50 times.For this number of learning times, which is necessary for comparison with other protocols, the most suitable of n for proposed protocol (LMNALA) occurs in ranges between 15 and 45, that with consideration to taking precision metric for nodes' selected transmission radius, the n value in experiments has been assumed to be equal to 30 and 45.The same results with the supposed parameters for proposed protocols in a Q-model environment are achieved, and again the most suitable value of noccurs in ranges between 15 and 45.The value of probability threshold for automata's actions in order to stop the learning phases is supposed to be equal to 0.9.However as experiments have shown too, the cost and consumed energy for learning automata's computations, against the output result that it achieves, is low.
With consideration of the simulation environment and network size, CLATC, RAA_2L,RAA_3L protocols [9] and the homogeneous case (HOM), and the proposed protocols (LMNALA) and (LMNALAQ) have been investigated from three points of view: mean transmission radius, average number of neighbours for each node (the network connectivity situation) and the mean residual energy, for n equal to 30 and 45, and the parameter value of a equal to 0.1 and the initial value for parameter b equal to 0.01 for the first proposed protocol(LMNALA); and for the other one, 0.1 for the initial value of parameter a,and the initial value of parameter(b low )is supposedto be equal to 0.01.The achieved results are averaged over running of the above protocols for 100 random different network configurations.Figure 2shows the typically achieved topology after running the first proposed protocol (LMNALA) for 300 nodes in one experiment in comparison with when all nodes equally have their maximum transmission range in homogeneous case, and Figure 3 shows the same verification for the proposed protocol (LMNALAQ)in the random dissemination of nodes.

A. Experiment 1
The goal of this experiment is to consider the average transmission range of network sensors for CLATC, RAA_2L, RAA_3L and HOM protocols and the proposed protocols (LMNALA) and (LMNALAQ).Whenever a node's selected transmissions radius is lower, its consumed energy will be lower too, and according to decreasing the number of neighbours, the probability of congestion will be lower too.Figure 4 shows sensor nodes' mean transmission radiuses for above presented protocols for different networks of different sizes, and for the achieved adequate sub range numbers (n) equal to 30(A) and 45(B).As it has shown, the proposed protocol has an approximately optimized functionality from this point into others.The HOM case has the most average transmissions radius and this is because all nodes in it have the same transmissions radius equal to R t .The RAA_3L protocol has a smaller average transmissions radius than the RAA_2L, because in RAA_3L every node can choose its own transmissions radius from three available transmissions ranges; while in RAA_2L protocol every node chooses its own transmissions radius from two available transmissions ranges.

B. Experiment 2
In this experiment the average number of neighbours' nodes in the network for CLATC, RAA_2L, RAA_3L and HOM protocols and the proposed protocols (LMNALA) and (LMNALAQ) has been investigated.The average number of neighbours' nodes for 6 above presented protocols for different networks of different sizes and for the achieved adequate sub range numbers (n) equal to 30(A) and 45(B) is depicted in Figure 5. Since the number of neighbours has direct effect on congestion between nodes, decreasing this parameter has very high importance.As it is observed here, the proposed protocols have the least average number of neighbors' in comparison with other protocols.of sub-range=45 As Figure 5 also shows that between the two proposed protocols, LMNALAQ for both number of sub ranges has the least average number of neighbours.

C. Experiment 3
In this experiment the average residual energy of each node in the network for CLATC, RAA_2L, RAA_3L and HOM protocolsand the proposed protocols (LMNALA)and (LMNALAQ) has been investigated.Average residual energy of networks' nodes for different networks of different sizes represented in Figure 6.As it has shown average residual energy of sensors in the network is very high and near to initial energy of nodes, which is one joule.In spite of the fact that the consumed energy of proposed protocol is approximately many times greater than that ofRAA_2L, RAA_3Lprotocols, because of existence of learning phase in the proposed protocol.But, the fact is that this consumed energy in comparison with total energy of every node is so low.Residual energy increment with increment of number of sensor nodes in network is because of the transmission range decrement in networks with higher density.With consideration of expressed literature, can deduce that evaluated protocols without consuming much more energy, give a proper topology.

D. Experiment 4
The goal of this experiment is to determine the number of proper sub ranges, which means, determining num. of nodes the better value for n, relating to iteration times of learning algorithm for network.Whatever the higher value for n to be selected, the transmission range would be divided into more sub ranges and it causes to more selections as radio range to be given to every node, and the amount precision of each node in selection would be increased.Meanwhile (R s -R w /n) as a coefficient, which is the same as C, has a direct effect on selected actions of learning automata (increment and decrement).In this experiment the proposed protocols (LMNALA) and (LMNALAQ) have been investigated for different values of n equal to 1, 5, 15,30,45,60,90,150,1000 from three points of view: mean transmission radius, average number of neighbours for each node, which indicates the network connectivity situation, and the mean residual energy for number of nodes in the network equal to 300, and maximum times of learning equal to 50 times.Results in Figures7,8,9have shown that for the number of learning times which is equal to 50, we have the best results in a mid range between 30 and 45.Then value equal to 15 was proper too, but we have left it for the reason of requiring higher precision.
Figure7.Average number of network nodes' neighbours for proposed protocol for 300 nodes and maximum learning times equal to 50 in different n numbers.

Figure8.Averagetransmission radius of network nodes for proposed protocol for 300 nodes and maximum learning times equal to 50 in different numbers
As a result, with increasing the number of selected sub ranges n; we should also increase the number of learning times; so that can get to desired conclusion; but this task is being along with more energy consumption.Also, results of Figures 6,7,8 indicate better functionality of LMNALAQ protocol from three points of view that are: mean transmission radius, average number of neighbours for each node (the network connectivity situation) and the mean residual energy; or in other words the Q-model learning environment is a better way for solving the proposed problem in this paper, which is selecting the most suitable transmission radius from transmission range between [R w ,R s ] for each node.

E. Experiment 5
The goal of this experiment is to investigate the selection probability of each of three assigned actions to learning automata in each run of learning iterations in one of the proposed protocol (LMNALA).One node is typically assumed and the probability of every selected action of its allocated learning automata is showed on it.According to achieved Figure 10, node with number 25 is randomly selected and after 12 stages of learning, its automata is converged to one of its actions which is not to change the radio transmission range.From this result, we can deduce that the convergence for actions of learning automata in LMNALAQ is also achievable and similar to LMNALA.

F. Experiment 6
The goal of this experiment is to investigate the connection of network and to determine the largest connected component of the network for RAA_2L, RAA_3L, HOM and proposed protocols (LMNALA and LMNALAQ).The largest normal connected components achieved by dividing the highest number of connected nodes by all nodes of network.Whatever the value of this computation to be higher, the network would have more connected nodes with ability of connecting to each other.The largest normal connected

VIII. CONCLUSIONS
In this paper neighbour-based topology control protocols which are based on learning automata are proposed.In these protocols, in spite of the fact that nodes by using learning automata and with negligible energy consumption, select proper transmission radius according to required conditions for preserving connectivity for themselves, But in return, they get to an adequate topology that to the extent possible, will have the lowest energy consumption for communication between nodes in the network's life time, if a tradeoff between the required precise values which we want to take as transmission ranges with learning times and energy consumption, would be taken into consideration.Also, the proposed protocols, LMNALA, LMNALAQ with two different numbers of sub ranges have been simulated and have been compared with other similar protocols.The results of comparisons have expressed the desired functionality of proposed protocols, especially LMNALAQ into other protocols.

Figure 1 .
Figure 1.Relationship between learning automata and its environment.

Figure 2 .Figure 3 .
Figure 2. Left) Maximum power graph with 300 nodes Right) Achieved graph of running the first proposed protocol (LMNALA)

Figure 4 .
Figure 4.Average transmission radius of network's nodes for CLATC, RAA_2L,RAA_3L, HOM and proposed protocols in different network sizes.A) with num. of sub-range=30 B)with num.of sub-range=45

Figure5.
Figure5.Average number of neighbours of network's nodes for CLATC, RAA_2L,RAA_3L, HOM and proposed protocols in different network sizes.A) with num. of sub-range=30 B)with num.of sub-range=45

Figure 6 .
Figure 6.Average residual energy of network's nodes for CLATC, RAA_2L,RAA_3L, HOM and proposed protocols in different network sizes with number of sub-range=30 and 45.

Figure 9 .
Figure 9. Average residual energy of network nodes for proposed protocol for 300 nodes and maximum learning times equal to 50 in different n numbers.

Figure 10 .
Figure 10.convergence the automata actions for node number 25 in the proposed protocol (LMNALA) component of the network for four mentioned protocols in different network sizes is depicted in Figure11.According to it, proposed protocols have the largest normal connected component to the extent of RAA_2L, RAA_3L protocols.Among all, LMNALAQ is along with better proportional functionality in preserving largest normal connected component and could decrease the average transmission range and the average number of neighbours' nodes of network sensors more than others.The achieved results are from an investigation of the network connectivity in running 100 different network configurations.

Figure11.
Figure11.The largest normal connected component for RAA_2L,RAA_3L, HOM and proposed protocols in different network sizes.